[issue10915] Make the PyGILState API compatible with multiple interpreters

[Maciej Szulik]

Change by Maciej Szulik :


--
nosy: +maciej.szulik

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-1021318: "PyThreadState_Next not thread safe".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Paulo Henrique Silva]

Change by Paulo Henrique Silva :


--
nosy: +phsilva

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Eric Snow]

Change by Eric Snow :


--
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Ronald Oussoren]

Change by Ronald Oussoren :


--
nosy: +ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Nick Coghlan]

Nick Coghlan  added the comment:

A more recent discussion of this on python-dev: 
https://mail.python.org/pipermail/python-dev/2019-January/156095.html

The situation there appears to be a case of "Hand off an OS level thread from 
the creating interpreter to a different subinterpreter. As far as I can tell, 
calling GILState_Ensure in such a thread will still acquire the GIL of the 
creating interpreter (or something equally nonsensical)."

It's a single-threaded application using subinterpreters, but all the callbacks 
from the NumPy code end up hitting the original interpreter that initialised 
the thread local state in the main thread.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Patch probably doesn't apply anymore.

--
stage: patch review -> needs patch
versions: +Python 3.8 -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Petr Viktorin]

Change by Petr Viktorin :


--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Roundup Robot]

Roundup Robot added the comment:

New changeset e590c632c9fa by Victor Stinner in branch 'default':
Add more checks on the GIL
https://hg.python.org/cpython/rev/e590c632c9fa

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow
versions: +Python 3.5, Python 3.6 -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5d078b0bae75 by Victor Stinner in branch 'default':
Issue #19787: PyThread_set_key_value() now always set the value
http://hg.python.org/cpython/rev/5d078b0bae75

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[STINNER Victor]

STINNER Victor added the comment:

See also issue #15751.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[STINNER Victor]

Changes by STINNER Victor victor.stin...@gmail.com:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Graham - the cases you describe are the things I was saying don't currently 
work in my post and wouldn't be helped by Antoine's patch. Your thoughts on how 
we could possibly make it work actually parallel mine (although there may be 
fun and games with making sure the respective GILs are acquired and released in 
an appropriate order when switching interpreters).

However, if a given OS thread is created by a subinterpreter via 
thread_PyThread_start_new_thread, then the thread bootstrapping process will 
copy the thread state from that subinterpreter rather than the main interpreter.

Accordingly, the only thing that I believe should currently work with 
subinterpreters is the naive use case you described earlier (i.e. a call out to 
an extension module from a Python created thread that later calls back in using 
the PyGILState API in the exact same thread should work even in the presence of 
multiple interpreters).

This is the use case that I believe Antoine's patch as it currently stands 
actively *breaks* by making the autoTLSkey interpreter dependent.

Regardless, I'm marking 10914 as a dependency of this one, as I don't think we 
should change anything in this area until we have some unit tests to properly 
define what does and doesn't work. If we're going to promote subinterpreters to 
a robust, fully supported feature we may as well do it right.

--
dependencies: +Python sub-interpreter test

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Graham - the cases you describe are the things I was saying don't
 currently work in my post and wouldn't be helped by Antoine's patch.
 Your thoughts on how we could possibly make it work actually parallel
 mine (although there may be fun and games with making sure the
 respective GILs are acquired and released in an appropriate order when
 switching interpreters).

There is only a single GIL, not one per interpreter. And this mustn't
change since some objects are shared and their reference counts
shouldn't be touched concurrently.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Good point - consider that comment revised to refer to the GIL acquisition 
counter in the thread state struct. It may just be a matter of having 
ThreadState_Swap complain loudly if the gilstate_counter isn't set to a value 
it knows how to handle.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 The bulk of use cases is going to be simple callbacks via the same
 thread that called out of Python in the first place. [...]
 This is what SWIG effectively does in its generated wrappers for
 callbacks.

Thanks for clarifying. I agree the TLS scheme would help in these cases.
There is still the question of what/who updates the TLS mapping; you are
proposing a new API call; an alternative is to do the mapping in e.g.
PyEval_SaveThread().

 The reality is that if you force a change on every single extension
 module doing callbacks into the interpreter without having the GIL
 first, you will never see people update their code as they will likely
 not care about this special use case. And so the whole point of adding
 the additional APIs will be wasted effort and have achieved nothing.

I'm not sure I care. If people don't want to use the new APIs on the
basis that they are slightly more complex, then it's their problem. The
Python C API tries not to be too cumbersome, but it cannot pretend to be
as transparent as a high-level API in a dynamic language.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

There's no point improving the multiple interpreter support if it doesn't help 
applications that are currently broken because of that lack of support.

I believe the patch as currently proposed actually makes things *worse*. With 
autoTLSkey as a static variable, all subinterpreters will use the same key to 
point into thread local storage (which is defined process-wide). This means 
they won't tread on each other's toes: the interpreter that creates a thread 
owns that thread. So Graham's simple use case *should already work*, as the 
creation of the thread by the subinterpreter will populate autoTLSkey with a 
valid thread state, which will then be used by calls back in to the GILState 
API.

Looking at 3.2, there appear to be two ways for an application to get itself 
into trouble:

1. Hand off an OS level thread from the creating interpreter to a different 
subinterpreter. As far as I can tell, calling GILState_Ensure in such a thread 
will still acquire the GIL of the creating interpreter (or something equally 
nonsensical). I may be misreading that though - this isn't exactly the easiest 
part of the code base to follow :)

2. Native (non-Python) threads will always have their temporary thread state 
created in the main interpreter unless the application takes steps to precreate 
a thread state using a different interpreter.

So, a new PyGILState_EnsureEx API may be beneficial regardless in order to help 
with part 2 of the problem, but I think it should be used solely as a way to 
override autoInterpreterState. autoTLSkey should be left alone so that a 
given OS level thread can only be owned by one interpreter at a time.

Further, there is no need for any function with access to a valid thread state 
(i.e. _PyGILState_NoteThreadState, as well as PyGILState_Release if autoTLSkey 
remains a static variable) to take an interpreter argument. These functions can 
identify the relevant interpreter from the interp field of the thread state.

TL;DR version:
- I agree the compatibility between multiple interpreters and the GILState API 
can be improved
- I agree a PyGILState_EnsureEx that takes an interpreter argument should be 
part of that solution.
- I *disagree* with making autoTLSkey interpreter specific, as it seems to me 
that will make the situation worse rather than better.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Added Mark Hammond to the nosy list, as the original author and implementor of 
PEP 311 (which added the GILState APIs).

Mark, since your PEP deliberately punted on multiple interpreter support, feel 
free to take yourself off the list again. If you spot any glaring errors in my 
post above it would be nice to know, though :)

--
nosy: +mhammond

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

Nick, I think you are making the wrong assumption that an external threads will 
only ever call into the same interpreter. This is not the case. In mod_wsgi and 
mod_python there is a pool of external threads that for distinct HTTP requests, 
delegated to a specific thread, can make calls into different interpreters. 
This is all fine so long as you ensure that for each thread, it uses a distinct 
thread state for that thread for each interpreter. In other words, you cant use 
the same thread state instance across multiple interpreters as it is bound to a 
specific interpreter.

This is because autoInterpreterState is always going to be set to the main 
interpreter. This means that when the thread is calling into a new sub 
interpreter it will either inherit via current GIL state API an existing thread 
state bound to the main interpreter, or if one is created, will still get bound 
to the main interpreter. As soon as you start using a thread state bound to one 
interpreter against another, problems start occurring.

After thinking about this all some more I believe now what is needed is a mix 
of the TLS idea for current interpreter state that I am suggesting and in part 
the extended GIL state functions that Antoine describes.

So, the TLS records what interpreter a thread is currently running against so 
that GIL state APIs work for existing unmodified extension modules. At the same 
time though, you still need a way of switching what interpreter a thread is 
running against. For the latter, various of the thread state related functions 
that exist already could do this automatically. In some cases you will still 
need the extended function for acquisition that Antoine suggested.

Consider a few scenarios of usage.

First off, when an external thread calls PyInterpreter_New(), it creates a new 
thread state object against that new sub interpreter automatically and returns 
it. With this new systems, it would also automatically update the TLS for the 
current thread to be that new interpreter also. That way when it calls into 
Python which then calls back out to code which releases the GIL and then calls 
back in through PyGILState_Ensure(), with no arguments, it will work. This 
obviously implies though that PyGILState_Ensure() makes use of the TLS for the 
interpreter being used and isn't hard wired to the main interpreter like it is 
now.

Second, consider some of the other API functions such as PyThreadState_Swap(). 
When passing it a non NULL pointer, you are giving it a thread state object 
which is already bound to an interpreter. It thus can also update the TLS for 
the interpreter automatically. If you pass it a NULL then it clears the TLS 
with all functions later that rely on that TLS asserting that it is not NULL 
when used. Another similar case where TLS can be auto updated is functions 
which clear/delete an interpreter state and leave GIL unlocked at the end. 
These also would clear the TLS.

So, it is possible that that no new API functions may be needed to manage the 
TLS for what interpreter is associated with the current thread, as I thought, 
as existing API functions can do that management themselves transparently.

The third and final scenario, and the one where the extended GIL state 
functions for Ensure is still required, is where code doesn't have the GIL as 
yet and wants to make a call into sub interpreter rather than the main 
interpreter, where it already has a pointer to the sub interpreter and nothing 
more. In this case the new PyGILState_EnsureEx() function is used, with the sub 
interpreter being passed as argument.

The beauty of existing API functions of PyThreadState_Swap() etc managing the 
TLS for the interpreter is that the only code that needs to change is the 
embedded systems which are creating and using multiple interpreters in the 
first place. In other words, mod_wsgi would need to change, with it simply 
replacing all the equivalent stuff it already has for doing what PyGILState_??? 
functions do now but against sub interpreters. If I am right, all extension 
modules that don't really care about whether sub interpreters are being used 
should work without modification.

Oh, and I also think you probably don't need PyGILState_ReleaseEx() if all made 
TLS aware, just the single PyGILState_EnsureEx() is needed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

New submission from Antoine Pitrou pit...@free.fr:

It should be relatively easy to devise a new PyGILState API with support for 
multiple interpreters. We just need two new functions (similar to the two 
existing ones) taking a PyInterpreterState* parameter; a TLS key can be added 
to the PyInterpreterState structure (instead of the current global TLS key).

It will be up to the caller to know which interpreter they want to hook into 
when calling these functions (which is application-dependent and is normally 
well-defined, e.g. when calling a Python callback, you should call it with the 
interpreter which was in use when registering the callback (i.e. 
``PyThreadState_Get()-interp``)).

--
components: Interpreter Core
messages: 126333
nosy: amaury.forgeotdarc, grahamd, loewis, pitrou
priority: normal
severity: normal
stage: needs patch
status: open
title: Make the PyGILState API compatible with multiple interpreters
type: feature request
versions: Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Here is a sketch, including conversion of ctypes to the new API.
Converting sqlite would require a bit more work.

--
keywords: +patch
Added file: http://bugs.python.org/file20416/gilstateinterp.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


Removed file: http://bugs.python.org/file20416/gilstateinterp.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
stage: needs patch - patch review
Added file: http://bugs.python.org/file20417/gilstateinterp.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

Can you please provide an example of what user would do and what changes 
existing extension modules would need to make?

When I looked at this exact problem some time back, I worked out that you 
probably only need a single new public API function. This would be something 
like PyInterpreterState_Swap().

By default stuff would work on the main interpreter, but if for a specific 
thread it wanted to operate in context of a different sub interpreter, would 
call PyInterpreterState_Swap() to indicate that. That would store in TLS 
outside of any existing data structures. Functions like existing 
PyGILState_Ensure()/PyGILState_Release() would then look up that TLS variable 
to know which interpreter they are working with.

Doing it this way meant that no C extension modules using PyGILState_??? 
functions would need to change at all, as what interpreter is being operated on 
dictated by who created the thread and initiated call in to Python interpreter.

You probably want validation checks to say that PyInterpreterState_Swap() can 
only be called when not GIL lock held.

It worries me that you are talking about new PyGILState_??? functions as that 
would suggest to me that extension modules would need to change to be aware of 
this stuff. That you are saying that sqlite needs changes is what makes me 
things the way you are going is a problem. It isn't practical to make SWIG 
change to use something other than PyGILState_Ensure()/PyGILState_Release(), it 
should be transparent and required no changes to existing C extensions.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 Can you please provide an example of what user would do and what
 changes existing extension modules would need to make?

The patch contains such a change for ctypes. It's quite simple actually.

 By default stuff would work on the main interpreter, but if for a
 specific thread it wanted to operate in context of a different sub
 interpreter, would call PyInterpreterState_Swap() to indicate that.
 That would store in TLS outside of any existing data structures.
 Functions like existing PyGILState_Ensure()/PyGILState_Release() would
 then look up that TLS variable to know which interpreter they are
 working with.

That sounds like an ugly hack to avoid passing the desired interpreter
state directly to PyGILState_Ensure()/PyGILState_Release().

Besides, it will only work if a thread always serves the same
sub-interpreter.

 Doing it this way meant that no C extension modules using
 PyGILState_??? functions would need to change at all, as what
 interpreter is being operated on dictated by who created the thread
 and initiated call in to Python interpreter.

Who would do that, if it's not the extensions in question?
who created the thread is often a third-party library (e.g. sqlite)
that has no notion of a Python interpreter. That's the whole point of
using the PyGILState_* API, really. So extensions *will* have to be
fixed.

 That you are saying that sqlite needs changes is what makes me things
 the way you are going is a problem. It isn't practical to make SWIG
 change to use something other than
 PyGILState_Ensure()/PyGILState_Release(), it should be transparent and
 required no changes to existing C extensions.

What does SWIG use them for?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

A TLS based approach would presumably allow an embedding application like 
mod_wsgi to tinker with the state of threads created by naive modules that are 
unaware of the existence of subinterpreters.

That said, I don't see anything that prevents us from pursuing a TLS based 
override for the existing PyGILState functions later if the simpler, more 
explicit approach proves inadequate. As it stands, the new explicit calls allow 
something like mod_wsgi to define its *own* TLS location for the interpreter 
that is currently handling callbacks into Python, then use SWIG to generate 
PyGILState_*Ex calls in callback wrappers that reference that TLS interpreter 
state.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 A TLS based approach would presumably allow an embedding application
 like mod_wsgi to tinker with the state of threads created by naive
 modules that are unaware of the existence of subinterpreters.

The question is how mod_wsgi could know about the existence of these
threads, let alone decide which subinterpreter an arbitrary OS thread
should belong to; only the extension module can safely tell.
And it becomes totally hopeless if those threads are actually *shared*
between subinterpreters, as might be the case with a 3rd-party library
managing its own helper threads (I don't know if that's the case with
sqlite).

IMO we should really promote clean APIs which allow solving the whole
problem, rather than devise an internal hack to try to improve things
slightly.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

The bulk of use cases is going to be simple callbacks via the same thread that 
called out of Python in the first place. Thus ultimately all it is doing is:

Py_BEGIN_ALLOW_THREADS

Call into some foreign C library.
C library wants to do a callback into Python.

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* Perform Python actions here. */
result = CallSomeFunction();
/* evaluate result or handle exception */

/* Release the thread. No Python API allowed beyond this point. */
PyGILState_Release(gstate);

More stuff in C library.
Return back into the C extension wrapper.

Py_END_ALLOW_THREADS

This is what SWIG effectively does in its generated wrappers for callbacks.

Using a TLS solution, all these modules that simply do this will now start 
working where as they currently usually deadlock or have other problems.

In your solution, all these modules would need to be modified to some how 
transfer information about the current interpreter into the callback which is 
called by the foreign C library and use new PyGILState_??? functions rather 
than the old.

I do accept that more complicated extension modules which create their own 
foreign threads and perform the call back into interpreter from that thread, or 
systems like mod_wsgi which have a persistent thread pool from which calls 
originate, will have to be modified, but this is the lessor use case from what 
I have seen.

Overall, it is an easy win if TLS is used because a lot of code wouldn't need 
to change. Some will, but expect that a lot of the common stuff like lxml for 
example wouldn't.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10915] Make the PyGILState API compatible with multiple interpreters

[Graham Dumpleton]

Graham Dumpleton graham.dumple...@gmail.com added the comment:

As to the comment:

IMO we should really promote clean APIs which allow solving the whole
problem, rather than devise an internal hack to try to improve things
slightly.

The reality is that if you force a change on every single extension module 
doing callbacks into the interpreter without having the GIL first, you will 
never see people update their code as they will likely not care about this 
special use case. And so the whole point of adding the additional APIs will be 
wasted effort and have achieved nothing.

The TLS solution means many modules will work without the authors having to do 
anything.

You therefore have to balance between what you perceive as a cleaner API and 
what is actually going to see a benefit without having to wait a half dozen 
years before people realise they should change their ways.

BTW, TLS is currently used for current thread state for simplified GIL API, why 
isn't that use of TLS a hack where as doing the same for interpreter is?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10915
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com