仅在运行 Cython 编译的 .py 时死锁，而不是从解释器运行时

编程问答 2022-05-29

问题描述

从解释器运行时，以下示例代码按预期运行。但是，当我对示例进行 cythonize 处理然后导入已编译的模块时，我遇到了死锁。

您知道这种行为的原因是什么吗？我可以以某种方式解决吗？是否可以实现超时，以便线程在一定时间后终止？

from concurrent import futures
import time

class StopFlag:
    def __init__(self):
        self._started = 0
    
    @property
    def started(self):
        return self._started
        
    def stop(self):
        self._started = 0
        
    def start(self):
        self._started = 1
        
    
def my_loop(stop_flag):
        total_records = 0

        print ("Started loop")
        # wait until measurement is running actively
        while not stop_flag.started:
            pass
        
        print ("Started measurement")
        # wait until measurement is over            
        while stop_flag.started:
            pass
        
        print ("Measurement over")


stop_flag = StopFlag()

with futures.ThreadPoolExecutor() as t:
    t.submit(my_loop,stop_flag)

    time.sleep(1)
    stop_flag.start()
    time.sleep(1)
    stop_flag.stop()

解释：

$ py -3.6 treading.py
Started loop
Started measurement
Measurement over

Cythonized：

$ cythonize -ai3 treading.py
Compiling C:\mydir\Trials\treading.py because it changed.
[1/1] Cythonizing C:\mydir\Trials\treading.py
running build_ext
building 'treading' extension
creating C:\mydir\Trials\tmpcu1r5lge\Release
creating C:\mydir\Trials\tmpcu1r5lge\Release\mydir
creating C:\mydir\Trials\tmpcu1r5lge\Release\mydir\Trials
C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MT "-Ic:\program files (x86)\microsoft visual studio\shared\python36_64\include" "-Ic:\program files (x86)\microsoft visual studio\shared\python36_64\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.28.29333\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.28.29333\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /TcC:\mydir\Trials\treading.c /FoC:\mydir\Trials\tmpcu1r5lge\Release\mydir\Trials\treading.obj
treading.c
C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /nodefaultlib:libucrt.lib ucrt.lib /DLL /MANIFEST:EMbed,ID=2 /MANIFESTUAC:NO "/LIBPATH:c:\program files (x86)\microsoft visual studio\shared\python36_64\libs" "/LIBPATH:c:\program files (x86)\microsoft visual studio\shared\python36_64\PCbuild\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.28.29333\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.28.29333\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_treading C:\mydir\Trials\tmpcu1r5lge\Release\mydir\Trials\treading.obj /OUT:C:\mydir\Trials\treading.cp36-win_amd64.pyd /IMPLIB:C:\mydir\Trials\tmpcu1r5lge\Release\mydir\Trials\treading.cp36-win_amd64.lib
   Bibliothek "C:\mydir\Trials\tmpcu1r5lge\Release\mydir\Trials\treading.cp36-win_amd64.lib" und Objekt "C:\mydir\Trials\tmpcu1r5lge\Release\mydir\Trials\treading.cp36-win_amd64.exp" werden erstellt.
Code wird generiert.
Codegenerierung ist abgeschlossen.

$ py -3.6 -c "import treading;"
Started loop
<< deadlock >>

这是future.submit调用的编译代码：

+38: with futures.ThreadPoolExecutor() as t:
+39:     t.submit(my_loop,stop_flag)
          __Pyx_GetModuleGlobalName(__pyx_t_2,__pyx_n_s_t); if (unlikely(!__pyx_t_2)) __PYX_ERR(0,39,__pyx_L6_error)
          __Pyx_GOTREF(__pyx_t_2);
          __pyx_t_3 = __Pyx_PyObject_GetAttrStr(__pyx_t_2,__pyx_n_s_submit); if (unlikely(!__pyx_t_3)) __PYX_ERR(0,__pyx_L6_error)
          __Pyx_GOTREF(__pyx_t_3);
          __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
          __Pyx_GetModuleGlobalName(__pyx_t_2,__pyx_n_s_my_loop); if (unlikely(!__pyx_t_2)) __PYX_ERR(0,__pyx_L6_error)
          __Pyx_GOTREF(__pyx_t_2);
          __Pyx_GetModuleGlobalName(__pyx_t_1,__pyx_n_s_stop_flag); if (unlikely(!__pyx_t_1)) __PYX_ERR(0,__pyx_L6_error)
          __Pyx_GOTREF(__pyx_t_1);
          __pyx_t_8 = PyTuple_New(2); if (unlikely(!__pyx_t_8)) __PYX_ERR(0,__pyx_L6_error)
          __Pyx_GOTREF(__pyx_t_8);
          __Pyx_GIVEREF(__pyx_t_2);
          PyTuple_SET_ITEM(__pyx_t_8,__pyx_t_2);
          __Pyx_GIVEREF(__pyx_t_1);
          PyTuple_SET_ITEM(__pyx_t_8,1,__pyx_t_1);
          __pyx_t_2 = 0;
          __pyx_t_1 = 0;
          __pyx_t_1 = __Pyx_PyObject_Call(__pyx_t_3,__pyx_t_8,NULL); if (unlikely(!__pyx_t_1)) __PYX_ERR(0,__pyx_L6_error)
          __Pyx_GOTREF(__pyx_t_1);
          __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
          __Pyx_DECREF(__pyx_t_8); __pyx_t_8 = 0;
          __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

解决方法

免责声明 - 我对 cython 一无所知，我今天第一次尝试。这个“答案”并没有真正回答问题。

我在 Linux 上测试了这个，问题也存在。代码的行为就像将使用协作多任务处理一样。如果我向 StopFlag.started 添加一个 time.sleep(0)，或者只打开一个文件（不读取它），那么它就可以工作。作为解决方法可能很有用。

编辑2： 似乎与 GIL 相关。下面的更改有帮助（至少在我的系统上）。

class StopFlag:
     @property
     def started(self):
+        with nogil:
+            pass
         return self._started

所以我猜，运行 my_loop 的线程一直持有 GIL（它在我的系统上使用 100% CPU），而主线程没有获得 GIL 能够真正改变 stop_flag 值。

要检查这一点，第二个更改：

with futures.ThreadPoolExecutor() as t:
    t.submit(my_loop,stop_flag)

    time.sleep(1)
    print("before stop_flag.start")
    stop_flag.start()
    print("after stop_flag.start")
    time.sleep(1)
    stop_flag.stop()

我仍然只在终端上：

python -c "import treading;"
Started loop

在t.submit之后，主线程确实执行了任何其他代码行，并且没有打印before stop_flag.start。

concurrent.futurescythonizedeadlockpython-3.x