Thanks to Renato, the solution was just... PyEval_InitThreads !
Thanks for your help !
Matthew, you're right, there is just one thread running at a time in Python.
But, if my understanding of the Python API is correct, more than one thread
can be registered to the Python Interpreter (PyThreadState). Calling
PyGILState_Ensure function creates a PyThreadState.
Next, the Python API will switch its current thread every 100 bytecodes
(this value is the default of sys.setcheckinterval function). There are
never two system threads running at the same time in the Python API, but at
least... they are alternated.
One proof is that PyGILState_Ensure is _not_ blocking the second thread
during the execution of the first thread in Python. It probably waits a few
microseconds, as I explain below, but the 2 threads enter the "critical"
Python part together. I checked that :
static int count = 0;
_gstate = PyGILState_Ensure();
if (count != 0)
// In my code, I use a thread-safe stream.
std::cout << "2 threads together" << std::endl;
count++;
// Interprets Python bytecodes with boost::python
count--;
PyGILState_Release(_gstate);
The message "2 threads together" appears. Without PyEval_InitThreads, it
crashes just after.
You are right about the locks, they are not needed.
Let's detail what happens :
a) Thread 1 : GIL_Ensure(New PyThread_State registered, GIL lock)
b) Thread 1 : Executing Python through the Interpreter | Thread 2 :
GIL_Ensure (waiting for GIL)
c) Thread 1 : Executing the 100th bytecode : Python Interpreter Releases the
GIL. Switch to Thread 2.
d) Thread 1 : waiting for GIL | Thread 2 : Executing Python
e) Thread 2 : 100th bytecode : Switch to Thread 1.
f) Thread 1 : Executing Python | Thread 2 : Waiting for GIL
etc.
--
BRIET Pascal
------------------------------
Remember of use PyEval_InitThreads on your module initialization.
Renato Araujo Oliveira Filho
------------------------------
Message: 3
Date: Tue, 2 Feb 2010 11:30:37 -0600
From: "Matthew Scouten (TT)" <matthew.scou...@tradingtechnologies.com>
To: "Development of Python/C++ integration" <cplusplus-sig@python.org>
Subject: Re: [C++-sig] PyGILState_Release with multithread
Message-ID: <32490dff7774554a85d65d23a9f0f9380c4de...@chiex01>
Content-Type: text/plain; charset="iso-8859-1"
First of all, the PyGILState_* functions are acquiring and releasing the
"Global Interpreter _Lock_" Your _pyMutex is redundant.
Here are the rules that govern the GIL:
1) Only one thread at a time can hold the GIL
2) Any thread that is touching python data, python code, or any part of
the 'terp must hold the GIL
The python part of your program WILL be single threaded. There is nothing
you can do about that.
From:
cplusplus-sig-bounces+matthew.scouten=tradingtechnologies....@python.org
[mailto:cplusplus-sig-bounces+matthew.scouten=tradingtechnologies....@python
.org] On Behalf Of Pascal Briet
Sent: Tuesday, February 02, 2010 5:46 AM
To: cplusplus-sig@python.org
Subject: [C++-sig] PyGILState_Release with multithread
Hello,
After a few days of headache, I think the best way is to share this
problematic...
I have :
- a main Python script
- a C module imported and called from Python (thanks to boost ::python)
- A dozen of threads created by a C function (boost ::thread)
- These threads, in a C main loop, call regularly a Python function.
Here is the critical part of the C code, with the Python GIL management :
boost::mutex _pyMutex;
PyGILState_STATE _gstate;
_pyMutex.lock();
_gstate = PyGILState_Ensure();
_pyMutex.unlock();
// Interprets Python bytecodes with boost::python
_pyMutex.lock();
PyGILState_Release(_gstate);
_pyMutex.unlock();
This part is multi-threaded, and it seems that the PyGILState_* functions do
not like it.
I have the following error with PyGILState_Release : ? This thread state
must be current when releasing ?
I think that when 2 threads enter PyGILState_Ensure, the second one is
considered as the current one. When the first one end... it is not current.
A quick look to the code of the Python API seems to confirm it.
So, the only solution I found is to lock the whole call to Python :
boost::mutex _pyMutex;
PyGILState_STATE _gstate;
_pyMutex.lock();
_gstate = PyGILState_Ensure();
// Interprets Python bytecodes with boost::python
PyGILState_Release(_gstate);
_pyMutex.unlock();
Everything is ok with this solution, except that... only one thread at a
time can call a Python method.
When 2 threads ask for a Python call, if the first one is a blocking
operation, the second one will wait... it is so bad !
Do you have any solution ?
Nb : I use Python 2.5.2
Thanks for your help.
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@python.org
http://mail.python.org/mailman/listinfo/cplusplus-sig
