[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: > New API: > - if a thread state exists, and the interpreter doesn't match the > requested one, fail with an error > - otherwise, use the requested interpreter That's not what I'm proposing. What I'm proposing is that the new API uses a per-interpreter TLS key (

[issue15751] Add PyGILState_SwitchInterpreter

Nick Coghlan added the comment: I'm not sure it makes sense to call this new API "PyGILState_EnsureEx". My concern is that the behaviour is quite different in the presence of an existing thread state: Ensure: - if a thread state exists, use that interpreter - otherwise, use the default interpr

[issue15751] Add PyGILState_SwitchInterpreter

Changes by Andrew Svetlov : -- nosy: +asvetlov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue15751] Add PyGILState_SwitchInterpreter

Graham Dumpleton added the comment: If you have a Ex version of Ensure, then if the interpreter argument is NULL, then it should assume main interpreter. That way the normal version of Ensure can just call PyGILState_EnsureEx(NULL). -- ___ Python tr

[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: > Those macros only work for general GIL releasing and pop straight > away, not for the case where released, calls into some non Python C > library, which then calls back into Python. I see, so you are right that this new API could be useful. However, we should

[issue15751] Add PyGILState_SwitchInterpreter

Graham Dumpleton added the comment: Those macros only work for general GIL releasing and pop straight away, not for the case where released, calls into some non Python C library, which then calls back into Python. My recollection is, and so unless they have changed it, SWIG generated calls us

[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: Le mardi 21 août 2012 à 22:14 +, Graham Dumpleton a écrit : > Any third party module then which uses the simplistic calling sequence > of calling PyGILState_Release() on exiting Python code and thence > within the same thread calling PyGILState_Ensure() when

[issue15751] Add PyGILState_SwitchInterpreter

Graham Dumpleton added the comment: In both embedded mode and daemon mode of mod_wsgi, albeit how thread pool is managed is different, there is a fixed number of threads with those being dedicated to handling web requests. On a request arriving next available thread from the thread pool handle

[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: > Umm, no. The whole point of the GILState API is that you can call it > from a thread which knows *nothing* about Python. No to what? Any sane callback API allows to pass it some user data, that user data can just as well include the pointer to the desired inte

[issue15751] Add PyGILState_SwitchInterpreter

Nick Coghlan added the comment: Umm, no. The whole point of the GILState API is that you can call it from a thread which knows *nothing* about Python. It will then use the *process global* state in the pystate.c statics to initialise that thread with a Python thread state. Currently, that thre

[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: > If we expose an official way to switch the destination of the > PyGILState APIs, then what it means is that mod_wsgi can, over the > lifecycle of a single process in the pool, *switch* the virtual host > and WSGI endpoint that process is handling by changing th

[issue15751] Add PyGILState_SwitchInterpreter

Nick Coghlan added the comment: s/slightly/completely/ (I believe my description of the core problem was right, but my description of why that problem exists was wrong - it actually has to do with the way mod_wsgi handles virtual hosts and endpoints) If we expose an official way to switch the

[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: > My understanding of the mod_wsgi architecture is that it uses > subinterpreters to maintain a persistent process, while still > providing a relatively pristine interpreter state to handle each new > request. I don't think that's true. On hg.python.org, the hgl

[issue15751] Add PyGILState_SwitchInterpreter

Nick Coghlan added the comment: Sorry, I mischaracterised the way mod_wsgi works slightly. However, my understanding is still that the scope of this particular fix is merely to allow all external threads to be redirected to a different subinterpreter at various times over the life of a process

[issue15751] Add PyGILState_SwitchInterpreter

Nick Coghlan added the comment: My understanding of the mod_wsgi architecture is that it uses subinterpreters to maintain a persistent process, while still providing a relatively pristine interpreter state to handle each new request. This means even when you're using multiple processes with a

[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: > Just as they do today, all externally created threads will still go to > *one* interpreter when they hit PyGILState_Ensure(). It's just that > interpreter won't be the main one. Uh but how does it solve the issue? (unless you create a mod_wsgi app with only a

[issue15751] Add PyGILState_SwitchInterpreter

Nick Coghlan added the comment: Just as they do today, all externally created threads will still go to *one* interpreter when they hit PyGILState_Ensure(). It's just that interpreter won't be the main one. Since the whole point of the PyGILState API is to support threads that don't have a pre

[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: > As I understand the situation, mod_wsgi doesn't need arbitrary > externally created threads to be able to call back into arbitrary > subinterpreters, it just needs to be able to direct externally created > threads in a process to a subinterpreter other than the

[issue15751] Add PyGILState_SwitchInterpreter

Nick Coghlan added the comment: Graham, even better would be if you could try the following combination: _PyGILState_Fini(); _PyGILState_Init(si, st); (where si and st are the interpreter state and thread state for the target subinterpreter) If a new PyGILState_SwitchInterpreter API is going

[issue15751] Add PyGILState_SwitchInterpreter

Nick Coghlan added the comment: It would twiddle the autoInterpreterState and autoTLSkey entries in the pystate.c global variables to point to a different subinterpreter. As I understand the situation, mod_wsgi doesn't need arbitrary externally created threads to be able to call back into arbi

[issue15751] Add PyGILState_SwitchInterpreter

Antoine Pitrou added the comment: > a function that mod_wsgi can call to set the interpreter used by the > GIL state APIs to implicitly create new thread states. How would it work? -- nosy: +pitrou ___ Python tracker

[issue15751] Add PyGILState_SwitchInterpreter

Graham Dumpleton added the comment: Just to clarify. One can still tell WSGI applications under mod_wsgi to run in the main interpreter and in that case modules using PyGILState* do work. By default though, sub interpreters are used as noted. The mechanism for forcing use of main interpreter i

[issue15751] Add PyGILState_SwitchInterpreter

New submission from Nick Coghlan: Currently, modules that use the PyGILState* APIs cannot be used with mod_wsgi, as mod_wsgi uses the subinterpreter support. Graham Dumpleton and I spent some time discussing this at PyCon AU 2012, and we believe that the incompatibility can be resolved with a