On Fri, 26 Feb 2010 11:05:44 +0100, Denis RIVIERE <denis.rivi...@cea.fr> wrote: > We have run into what we think to be more or less a bug in PyQt4, in > QProcess, > somewhat indirectly. > QProcess sometimes hangsup in threaded PyQt applications, at least on Linux > > systems. > The situation is the following: > - A PyQt program is using two or more threads (say, python threads) > - one of the threads is using a QProcess (whatever thread), while the other > is > using python (of course). > > We think we have understood what is going on: > > Internally, QProcess forks then probably uses a kind of exec(). > But the child forked process sometimes gets locked on the python GIL lock. > Then the calling process waits for the child, and also gets hung on a > select() > call. > > Actually, when forking a multi-threaded process, only the forking thread is > > duplicated: so the child process has only 1 thread. But the locks, mutexes > and other threading items have kept the state they had in their parent > process: some are locked, but the other threads which would normally > release > them do not exist in the child. > Here, the python GIL happens to be locked by another thread in the parent
> process, and the child process seems to try to lock the GIL too. We guess > QProcess calls a virtual method in the forked process, which is overloaded > by > SIP/PyQt, which in turn calls the Python API, and finally gets hung trying > to > lock the GIL. > We have never encountered this problem using PyQt3, maybe because QProcess > or > its PyQt binding may be implemented differently. > > This threading + fork problem is a known difficulty in fork(), explained in > > the linux manpages for instance. > > We have 2 possible solutions for this problem: > 1. Using pthreads, it is possible to define handlers functions that can > take > care of locking/unlocking the needed locks whenever fork() is called, both > in > the parent and child processes: this is done using the pthread_atfork() > function. Typically PyQt may define such handlers to take care of the GIL > locking when fork() is used. However we don't know if such a mechanism > exists > on non-pthread systems (Windows?). We also don't really know if the problem > > exists on Windows; possibly not because Windows doesn't use fork() but > probably rather a spawn equivalent (?). > 2. The problem actually happens because the python API gets called from the > > forked process. Normally, in C++ Qt, the QProcess doesn't trigger such a > locking problem. However the SIP binding defines a subclass which traps all > > virtual methods calls and redirects them to python. So if we are not using > a > sip-inherited QProcess class, the problem should not happen. We have made > the > test, making a sip-bound factory function which instantiates a QProcess > from > C++, then returns it to the pyton layer. This way no inherited class is > used, > and the problem is avoided. And it actually works. > > Would it be difficult to have solution 1 included in the standard PyQt ? Unless you can provide me with a test case you'll have to experiment... Try adding... pthread_atfork(NULL, NULL, PyOS_AfterFork); ...after the call to PyEval_InitThreads() in siplib.c. Phil _______________________________________________ PyQt mailing list PyQt@riverbankcomputing.com http://www.riverbankcomputing.com/mailman/listinfo/pyqt