Another link that fills in some gaps and finally helped me make this work:
http://www.codevate.com/blog/7-concurrency-with-embedded-python-in-a-multi-threaded-c-application In particular, I found that PyGILState_Ensure/PyGILState_Release as described in the Python docs is not sufficient - as described in that blog link, I had to a) obtain PyInterpreterState from the first thread where Py_Initialize() was called b) when each worker thread starts, call PyThreadState_New(mInterpreterState) and save the result in a thread local mPyThreadState c) use the mPyThreadState with PyEval_RestoreThread and PyEval_SaveThread before and after calling Python methods Is this a bug in PyGILState_Ensure or is it a deficiency in the documentation? I also found one bug in my own code, although that was not related to the problem just described with PyGILState_Ensure and I had to fix both problems to make it work. Specifically, the PyWorkerThread constructor was taking an object argument when it should have taken a reference argument and this was creating an invalid Py::Callable member in my worker. On 18/12/13 00:19, Daniel Pocock wrote: > > I've successfully embedded Python for a single thread > > I tried to extend the implementation for multiple threads (a worker > thread scenario) and I'm encountering either deadlocks or seg faults > depending upon how I got about it. > > There seems to be some inconsistency between what is covered in the docs > here: > > http://docs.python.org/2/c-api/init.html#non-python-created-threads > > and the experiences of other users trying the same thing, e.g. > > http://bugs.python.org/issue19576 > http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads > > Can anybody comment on the situation, in particular, > > Is the non-python-created-threads documentation accurate for v2.7? > > If a main thread does things like importing a module and obtaining a > reference to a Python method, can those things be used by other C++ > threads or do they have to repeat those lookups? > > Is there any logic that needs to be executed once only as each thread is > started? (the doc suggests just PyGILState_Ensure/PyGILState_Release > each time a thread accesses Python methods - is there anything else?) > > Given the bug 19576, what is the most portable way to code this to work > reliably on unfixed Python versions? (e.g. should users always > explicitly call PyEval_InitThreads() in their main thread or worker > threads or both?) > > > > > Here is my actual source code: > > https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/ > > (see example.py for a trivial example of what it does) > > The problem that I encounter: > > - the init stuff runs fine in PyRoutePlugin.cxx, > it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule > and looks up the "provide_route" method in the module > it creates a PyRouteWorker object, > giving it a reference to "provide_route" > it creates a thread pool to run the worker > > - the PyRouteWorker::process() method is invoked in one of those threads > > - it crashes when trying to call the "provide_route" method > PyRouteWorker.cxx: > routes = mAction.apply(args); > > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7ffff30b8700 (LWP 23965)] > 0x00007ffff3d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 > (gdb) bt > #0 0x00007ffff3d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0 > #1 0x00007ffff3d6b647 in PyEval_CallObjectWithKeywords () > from /usr/lib/libpython2.7.so.1.0 > #2 0x00007ffff414885a in apply (args=..., this=<optimized out>) > at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215 > #3 repro::PyRouteWorker::process (this=0x6f00a0, msg=<optimized out>) > at PyRouteWorker.cxx:98 > #4 0x00007ffff7b879e1 in repro::WorkerThread::thread (this=0x68e110) > at WorkerThread.cxx:36 > #5 0x00007ffff70b7a2f in threadIfThreadWrapper (threadParm=<optimized out>) > at ThreadIf.cxx:51 > #6 0x00007ffff65ffb50 in start_thread (arg=<optimized out>) > at pthread_create.c:304 > #7 0x00007ffff5999a7d in clone () > at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 > #8 0x0000000000000000 in ?? () > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/daniel%40pocock.com.au _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com