[issue32280] Expose `_PyRuntime` through a section name
Eric Snow added the comment: (Sorry for the delay!) Is the need for a named section due to the fact that _PyRuntime is part of the "internal" C-API (hidden behind the Py_BUILD_CORE macro)? I.E. would it help to build with Py_BUILD_CORE? Regarding the GIL: The plan for 3.9 is to have one GIL per interpreter, so the relevant struct would be PyInterpreterState (a part of the public C-API). That would be accessible at runtime via the fields of _PyRuntime.interpreters. Regarding what I said before about multiple runtimes per process: Honestly, I've yet to see any use case that would justify providing multi-runtime support. However, there are code-health benefits to keeping *all* runtime state out of static globals. So eliminating the _PyRuntime static variable is still a realistic possibility (for 3.9 or maybe later). If that happens then there would have to be a new embedding C-API oriented around (opaque) PyRuntimestate pointers. See PEP 432 and 587 for ongoing work to improve the embedding experience (for runtime initialization), including what the successors to Py_Initialize() look like. -- versions: +Python 3.9 -Python 3.8 ___ Python tracker <https://bugs.python.org/issue32280> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34651] Disallow fork in a subinterpreter.
Eric Snow added the comment: FYI, I plan on looking into this either today or next Friday. -- ___ Python tracker <https://bugs.python.org/issue34651> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12857] Expose called function on frame object
Change by Eric Snow : Removed file: https://bugs.python.org/file48418/122.pdf ___ Python tracker <https://bugs.python.org/issue12857> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37186] Everyone uses GIL wrong! = DEADLOCK
Eric Snow added the comment: The situation with subinterpreters and the gilstate API is a known problem (see issues #10915 and #15751). We plan on fixing that in the near future for 3.9 (and probably for 3.8). Feel free to chime in on those earlier issues. Note that the subinterpreter experience is going to be much improved with the 3.9 release. You also mentioned an issue with PyEval_AcquireThread(). If you don't think this is directly related to the gilstate API (I suspect it is) then please open a new issue for that. -- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> Make the PyGILState API compatible with multiple interpreters ___ Python tracker <https://bugs.python.org/issue37186> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15751] Support subinterpreters in the GIL state API
Change by Eric Snow : -- versions: +Python 3.8, Python 3.9 -Python 3.6 ___ Python tracker <https://bugs.python.org/issue15751> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10915] Make the PyGILState API compatible with multiple interpreters
Change by Eric Snow : -- versions: +Python 3.9 ___ Python tracker <https://bugs.python.org/issue10915> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: Also, thanks for the suggestions, Victor! -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: Well, for this PR I actually disabled the execution of pending calls during runtime finialization. I'd hoped it would help, but it did not. :( Regardless, I'll need to look even more closely at what is different during runtime finalization with my PR. -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: I had really hoped to get this in for 3.8, as I may need it. However, realistically I doubt it will take just a few minutes to resolve this. With the beta 1 targeted for today it makes sense to revert my change. :( (I may still try to convince our fearless RM to allow it in beta 2, but we'll see.) -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: FWIW, the failures before were in test_io and test_multiprocessing_spawn, not test_asyncio. -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: Please wait on reverting. I'm going to investigate. @koobs, yeah, I'll send you a key. Thanks! :) -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: FYI, after merging that PR I realized that the COMPUTE_EVAL_BREAKER macro isn't quite right. While the following scenario worked before, now it doesn't: 1. interpreter A: _PyEval_AddPendingCall() causes the global eval breaker to be set 2. interpreter B: the next pass through the eval loop uses COMPUTE_EVAL_BREAKER; it has no pending calls so the global eval breaker is unset 3. interpreter A: the next pass through the eval loop does not run the pending call because the eval breaker is no longer set This really isn't a problem because the eval breaker is triggered for the GIL pretty frequently. Furthermore, it won't be a problem once the GIL is per-interpreter (at which point we will switch to a per-interpreter eval breaker). If it is important enough then I can fix it. I even wrote up a solution. [1] However, I'd rather leave it alone (hence no PR). The alternatives are more complicated and the situation should be relatively short-lived. FWIW, in addition to the solution I mentioned above, I tried a few other ways: * have a per-interpreter eval breaker in addition to the global one * have only a per-interpreter eval breaker (the ultimate objective) * consolidate the pending calls for every interpreter every time UNSIGNAL_PENDING_CALLS and UNSIGNAL_ASYNC_EXC are used However, each has performance penalties while the branch I created does not. I try to be really careful when it comes to the performance of the eval loop. :) [1] https://github.com/ericsnowcurrently/cpython/tree/eval-breaker-shared -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: So far so good. :) I'll keep an eye on things and if the buildbots are still happy then I'll add back _PyEval_FinishPendingCalls() in _Py_FinalizeEx() in a few days. -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: New changeset 6a150bcaeb190d1731b38ab9c7a5d1a352847ddc by Eric Snow in branch 'master': bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-13714) https://github.com/python/cpython/commit/6a150bcaeb190d1731b38ab9c7a5d1a352847ddc -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: I've made a few tweaks and Victor did some cleanup, so I'm going to try the PR again. At first I'm also going to disable the _PyEval_FinishPendingCalls() call in _Py_FinalizeEx() and then enable it is a separate PR. Also, I've opened bpo-37127 specifically to track the problem with _PyEval_FinishPendingCalls() during _Py_FinalizeEx(), to ensure that the disabled call doesn't slip through the cracks. :) -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: That's really helpful, Pavel! Thanks for investigating so thoroughly. I'm going to double check all the places I've made the assumption that "tstate" isn't NULL and likewise for "tstate->interp". Is there an issue open for the bug in multiprocessing? If not, would you mind opening one? -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37127] Handling pending calls during runtime finalization may cause problems.
Eric Snow added the comment: Also, someone did manage to investigate and identify a likely cause: https://bugs.python.org/issue33608#msg342791 -- ___ Python tracker <https://bugs.python.org/issue37127> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37127] Handling pending calls during runtime finalization may cause problems.
New submission from Eric Snow : In Python/lifecycle.c (Py_FinalizeEx) we call _Py_FinishPendingCalls(), right after we stop all non-daemon Python threads but before we've actually started finalizing the runtime state. That call looks for any remaining pending calls (for the main interpreter) and runs them. There's some evidence of a bug there. In bpo-33608 I moved the pending calls to per-interpreter state. We saw failures (sometimes sporadic) on a few buildbots (e.g. FreeBSD) during runtime finalization. However, nearly all of the buildbots were fine, so it may be a question of architecture or slow hardware. See bpo-33608 for details on the failures. There are a number of possibilities, but it's been tricky reproducing the problem in order to investigate. Here are some theories: * daemon threads (a known weak point in runtime finalization) block pending calls from happening until some time after portions of the runtime have already been cleaned up * there's a race that causes the pending calls machinery to get caught in some sort infinite loop (e.g. a pending call fails and gets re-queued) * a corner case in the pending calls logic that triggers only during finalization Here are some other points to consider: * do we have the same problem during subinterpreter finalization (Py_EndInterpreter() rather than runtime finalization)? * perhaps the problem extends beyond finalization, but the conditions are more likely there * the change for bpo-33608 could have introduced the bug rather that exposing an existing one -- assignee: eric.snow components: Interpreter Core messages: 344202 nosy: eric.snow priority: normal severity: normal stage: test needed status: open title: Handling pending calls during runtime finalization may cause problems. type: crash ___ Python tracker <https://bugs.python.org/issue37127> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36818] Add PyInterpreterState.runtime.
Change by Eric Snow : -- resolution: rejected -> fixed ___ Python tracker <https://bugs.python.org/issue36818> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Change by Eric Snow : -- pull_requests: +13601 pull_request: https://github.com/python/cpython/pull/13714 ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36818] Add PyInterpreterState.runtime.
Eric Snow added the comment: New changeset 396e0a8d9dc65453cb9d53500d0a620602656cfe by Eric Snow in branch 'master': bpo-36818: Add PyInterpreterState.runtime field. (gh-13129) https://github.com/python/cpython/commit/396e0a8d9dc65453cb9d53500d0a620602656cfe -- ___ Python tracker <https://bugs.python.org/issue36818> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36839] Support the buffer protocol in code objects
Eric Snow added the comment: FWIW, I don't see the problem with supporting any read-only "buffer" object, rather than just bytes objects, for the string of bytes in a code object. That's all that Dino is proposing. The change is not invasive and solves a real need. The additional maintenance burden is negligible. Furthermore, the accommodation makes sense conceptually. -- ___ Python tracker <https://bugs.python.org/issue36839> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37088] Add a way to schedule a function to be called from the main thread
Eric Snow added the comment: If this is about signals, it isn't enough just to run on the main thread. It also has to be the main interpreter. -- ___ Python tracker <https://bugs.python.org/issue37088> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37088] Add a way to schedule a function to be called from the main thread
Eric Snow added the comment: > Well, it's absolutely the same mechanism that signal handlers use. Antoine changed signals a while back so they no longer use the pending calls machinery. -- ___ Python tracker <https://bugs.python.org/issue37088> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37088] Add a way to schedule a function to be called from the main thread
Eric Snow added the comment:
Note that I'm working on making pending calls per-interpreter (see issue #33608
and https://github.com/python/cpython/pull/12360 (since reverted)).
As to exposing Py_AddPendingCall() as sys.addpendingcall, that might be opening
a can of worms. Injecting code into the eval loop at some arbitrary ("soon")
future time requires care and the code isn't well exercised historically (much
like subinterpreters). By making it easier to use the pending calls API (e.g.
from Python code) we may be introducing an attractive nuisance. It also adds
burden on other Python implementations.
My point is, let's think this through before adding sys.addpendingcall(). :)
Is there another way this could be done that doesn't open a can of worms?
Also, at the very least it should probably be a "private" function (i.e
sys._addpendingcall).
--
nosy: +eric.snow
___
Python tracker
<https://bugs.python.org/issue37088>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] Global C variables are a problem.
Change by Eric Snow : -- pull_requests: +13445 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24932] Use proper command line parsing in _testembed
Change by Eric Snow : -- nosy: +vstinner ___ Python tracker <https://bugs.python.org/issue24932> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] Global C variables are a problem.
Change by Eric Snow : -- keywords: +patch pull_requests: +13283 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] Global C variables are a problem.
Eric Snow added the comment: Also, Tools/c-globals/ignored-globals.txt is a bit out of date (some vars have been removed, renamed, or moved to another file). That should get cleaned up. It might also make sense to update check-c-globals.py to verify that all variables in ignored-globals.txt actually exist. -- ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36877] [meta] Move fields from _PyRuntimeState to PyInterpreterState.
Eric Snow added the comment: In conjunction with #36876, there are a bunch of currently ignored globals (in Tools/c-globals/ignored-globals.txt) that should actually be moved to per-interpreter runtime state. This mostly applies to any globals that point to one or more objects, since objects must not be shared by interpreters. -- ___ Python tracker <https://bugs.python.org/issue36877> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36877] [meta] Move fields from _PyRuntimeState to PyInterpreterState.
Eric Snow added the comment: FYI, I've already started some of this work: * #36737 warnings * #36854 gc * #33608 pending calls * #10915 & #15751 gilstate Other bits I'm planning on: * the rest of the global "ceval" state * the memory allocators * the GIL Note that, to make the GIL per-interpreter, we can't have any remaining runtime state shared by interpreters (unless it is protected by its own locks). -- ___ Python tracker <https://bugs.python.org/issue36877> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36877] [meta] Move fields from _PyRuntimeState to PyInterpreterState.
New submission from Eric Snow : We have quite a bit of global state the runtime that effectively breaks the isolation between interpreters. Some of it exists as "global" C variables (see #36876) and the rest as fields on _PyRuntimeState. The offending state should be moved to PyInterpreterState. See Include/internal/pycore_pystate.h for the _PyRuntimeState and PyInterpreterState structs. -- assignee: eric.snow components: Interpreter Core messages: 342120 nosy: eric.snow, vstinner priority: normal severity: normal status: open title: [meta] Move fields from _PyRuntimeState to PyInterpreterState. versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36877> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] Global C variables are a problem.
New submission from Eric Snow : We still have a bunch of "global" C variables (static globals, static locals, maybe thread-local storage) in our code-base that break the isolation between interpreters. I added Tools/c-globals/check-c-globals.py a while back to help identify such variables, however more have crept in. I also did not take static locals or thread-locals into account. To address the above, we should do the following: 1. update check-c-globals.py to identify static locals (and thread-locals) 2. deal with any identified globals * move them to _PyRuntimeState (or thread-locals to PyThreadState, etc.) * ignore them by adding them to Tools/c-globals/ignored-globals.txt 3. add check-c-globals.py to "make check" 4. (if "make check" isn't already there), ensure check-c-globals.py is run at some point in CI Separately, we should move fields out of _PyRuntimeState into PyInterpreterState wherever possible. That can also be done at step 2 if it's not too much work. -- assignee: eric.snow components: Interpreter Core messages: 342119 nosy: eric.snow, vstinner priority: normal severity: normal stage: needs patch status: open title: Global C variables are a problem. type: behavior versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36737] Warnings operate out of global runtime state.
Eric Snow added the comment: New changeset 86ea58149c3e83f402cecd17e6a536865fb06ce1 by Eric Snow in branch 'master': bpo-36737: Use the module state C-API for warnings. (gh-13159) https://github.com/python/cpython/commit/86ea58149c3e83f402cecd17e6a536865fb06ce1 -- ___ Python tracker <https://bugs.python.org/issue36737> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36737] Warnings operate out of global runtime state.
Change by Eric Snow : -- assignee: -> eric.snow resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36737> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36854] GC operates out of global runtime state.
Change by Eric Snow : -- keywords: +patch pull_requests: +13130 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36854> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24554] GC should happen when a subinterpreter is destroyed
Eric Snow added the comment: FYI, issue #36854 is about moving GC runtime state from _PyRuntimeState to PyInterpreterState. However, that doesn't trigger any collection when the interpreter is finalized. So there is more to be done here. -- ___ Python tracker <https://bugs.python.org/issue24554> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36854] GC operates out of global runtime state.
New submission from Eric Snow : (also see #24554) We need to move GC state from _PyRuntimeState to PyInterpreterState. -- assignee: eric.snow components: Interpreter Core messages: 341911 nosy: eric.snow, pablogsal priority: normal severity: normal status: open title: GC operates out of global runtime state. versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36854> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36737] Warnings operate out of global runtime state.
Change by Eric Snow : -- keywords: +patch pull_requests: +13078 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue36737> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36835] Move the warnings runtime state into per-interpreter state.
Eric Snow added the comment: dupe of #36737 -- resolution: -> duplicate stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36835> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36835] Move the warnings runtime state into per-interpreter state.
Change by Eric Snow : -- keywords: +patch pull_requests: +13076 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue36835> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36835] Move the warnings runtime state into per-interpreter state.
New submission from Eric Snow : Currently the warnings module uses runtime-global state (PyRuntimeState.warnings). That should be moved down to per-interpreter state. There are three possible places: 1. the module's "module state" 2. the module's __dict__ 3. PyInterpreterState.warnings (new) I have a patch for the first option. -- assignee: eric.snow components: Interpreter Core messages: 341763 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Move the warnings runtime state into per-interpreter state. versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36835> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36818] Add PyInterpreterState.runtime.
New submission from Eric Snow : Currently we use the _PyRuntime static global to access the runtime state in various places. At the same time, in thread contexts we get access to the thread state from Thread-Local Storage and the interpreter state by indirection from there. We should do the same for the runtime state instead of using the global directly. My plan is to add a PyInterpreterState.runtime field. It can then be used in the same way we use PyThreadState.interp to access the interpreter state. -- assignee: eric.snow components: Interpreter Core messages: 341595 nosy: eric.snow, vstinner priority: normal severity: normal stage: patch review status: open title: Add PyInterpreterState.runtime. versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36818> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36818] Add PyInterpreterState.runtime.
Change by Eric Snow : -- keywords: +patch pull_requests: +13042 ___ Python tracker <https://bugs.python.org/issue36818> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36809] Crash for test test_importlib
Change by Eric Snow : -- nosy: +brett.cannon, eric.snow ___ Python tracker <https://bugs.python.org/issue36809> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36772] Let lru_cache be used as a decorator with no arguments
Eric Snow added the comment: As to the issue of positional vs. keyword arguments, keyword arguments make the implementation easier in some cases. Otherwise I haven't seen positional arguments cause much of a problem. -- ___ Python tracker <https://bugs.python.org/issue36772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36772] Let lru_cache be used as a decorator with no arguments
Eric Snow added the comment: FWIW, I've followed this pattern (one function is both decorator and factory) in my own code for quite a while. I've never found it confusing nor has anyone else (that I'm aware) that has used those decorators. One reason I've done decorators this way is because the empty parentheses are a visual distraction to readers. They also imply to readers that more is going on than really is. So I'm in favor of Raymond's plan. -- ___ Python tracker <https://bugs.python.org/issue36772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36737] Warnings operate out of global runtime state.
Eric Snow added the comment: Good point. Also, the whole idea of inheriting things (settings, some copied objects, etc.) into subinterpreters is interesting. My initial reaction is that folks would appreciate that feature, at least for a handful of things. It's not critical, but is worth adding to the list of deferred ideas in PEP 554. :) -- ___ Python tracker <https://bugs.python.org/issue36737> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable
Eric Snow added the comment: FWIW, I don't mean to side-track this issue. If we want to have any further discussion about broader solutions then let's take this to capi-sig. In fact, I've started a thread there. I'd post the link, but I think it got stuck in moderation. :) -- ___ Python tracker <https://bugs.python.org/issue36710> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable
Eric Snow added the comment:
I don't think this change is the right way to go (yet), but something related
might be. First, let's be clear on the status quo for CPython. (This has
gotten long, but I want to be clear.)
Status Quo
For simplicity sake, let's say nearly all the code operates relative to the 3
levels of runtime state:
* global - _PyRuntimeState
* interpreter - PyInterpreterState
* thread - PyThreadState
Furthermore, there are 3 groups of functions in the C-API:
* context-sensitive - operate relative to the current Python thread
* runtime-dependent - operate relative to some part of the runtime state,
regardless of thread
* runtime-independent - have nothing to do with CPython's runtime state
Most of the C-API is context-sensitive. A small portion is runtime-dependent.
A handful of functions are runtime-independent (effectively otherwise stateless
helper functions that only happen to be part of the C-API).
Each context-sensitive function relies on there being a "runtime context" it
can use relative to the current OS thread. That context consists of the
current (i.e. active) PyThreadState, the corresponding PyInterpreterState, and
the global _PyRuntimeState. That context is derived from data in TSS (see
caveats below). This group includes most of the C-API.
Each runtime-dependent function operates against one or more runtime state
target, regardless of the current thread context (or even if there isn't one).
The target state (e.g. PyInterpreterState) is always passed explicitly. Again,
this is only a small portion of the C-API.
Caveats:
* for context-sensitive functions, we get the global runtime state from the
global C variable (_PyRuntime) rather than via the implicit thread context
* for some of the runtime-dependent functions that target _PyRuntimeState, we
rely on the global C variable
All of this is the pattern we use currently. Using TSS to identify the
implicit runtime context has certain benefits and costs:
benefits:
* sticking with the status quo means no backward incompatibility for existing
C-extension code
* easier to distinguish the context-sensitive functions from the
runtime-dependent ones
* (debatable) callers don't have to track, nor pass through, an extra argument
costs:
* extra complexity in keeping TSS correct
* makes the C-API bigger (extra macros, etc.)
Alternative
=
For every context-sensitive function we could add a new first parameter,
"context", that provides the runtime context to use. That would be something
like this:
struct {
PyThreadState *tstate;
...
} PyRuntimeContext;
The interpreter state and global runtime state would still be accessible via
the same indirection we have now.
Taking this alternative would eliminate the previous costs. Having a
consistent "PyRuntimeContext *context" first parameter would maintain the easy
distinction from runtime-dependent functions. Asking callers to pass in the
context explicitly is probably better regardless. As to backward
compatibility, we could maintain a shim to bridge between the old way and the
new.
About the C-global _PyRuntime
==
Currently the global runtime state (_PyRuntimeState) is stored in a static
global C variable, _PyRuntime. I added it at the time I consolidated many of
the existing C globals into a single struct. Having a C global makes it easy
to do the wrong thing, so it may be good to do something else.
That would mean allocating a _PyRuntimeState on the heap early in startup and
pass that around where needed. I expect that would not have any meaningful
performance penalty. It would probably also simplify some of the code we
currently use to manage _PyRuntime correctly.
As a bonus, this would be important if we decided that
multiple-runtimes-per-process were a desirable thing. That's a neat idea,
though I don't see a need currently. So on its own it's not really a
justification for dropping a static _PyRuntime. :) However, I think the other
reasons are enough.
Conclusions
This issue has a specific objective that I think is premature. We have an
existing pattern and we should stick with that until we decide to change to a
new pattern. That said, a few things should get corrected and we should
investigate alternative patterns for the context-sensitive C-API.
As to getting rid of the _PyRuntime global variable in favor of putting it on
the heap, I'm not opposed. However, doing so should probably be handled in a
separate issue.
Here are my thoughts on actionable items:
1. look for a better pattern for the context-sensitive C-API
2. clearly document which of the 3 groups each C-API function belongs to
3. add a "runtime" field to the PyInterpreterState pointing to the parent
_PyRuntimeState
4. (maybe) add a _PyRuntimeState_GET() macro, a la PyThreadState_GET()
5. for cont
[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable
Eric Snow added the comment: FWIW, PEP 554 is part of a larger project that I've been working on (slowly) for several years now. [1] The concrete objective is to leverage subinterpreters as the mechanism by which we can achieve multi-core parallelism in Python code. Moving the GIL (and some other parts of _PyRuntimeState, as Victor indicated) down to per-interpreter state is essential to that. However, I don't thing making _PyRuntime a per-interpreter thing is right. The runtime holds the set of interpreters, as well as any state state shared by the interpreters. Also, to be clear, the status quo is not a problem for me, so make sure I'm not used as the justification for the change (thoughtful as that is of Victor). :) [1] https://github.com/ericsnowcurrently/multi-core-python -- ___ Python tracker <https://bugs.python.org/issue36710> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36737] Warnings operate out of global runtime state.
New submission from Eric Snow : (See Include/internal/pycore_warnings.h and Python/_warnings.c.) The warnings module's state (filters, default action, etc.) is currently stored at the level of the global runtime. That's a problem for the following reasons: * Python objects are getting stored in _PyRuntimeState * it breaks the isolation of behavior between interpreters * objects are leaking between interpreters * importing the module in a subinterpreter effectively resets the module's state While those are all a problem in a future where interpreters don't share the GIL, that last one is a problem right now for people using subinterpreters. One of the following should happen: * move warnings state down to PyInterpreterState * move warnings state into PyInterpreterState.dict * use the module-state API (PEP 3121) * just work out of the module's __dict__ I could also see use cases for *also* configuring warnings process-wide but that could be handled separately if actually desired. -- components: Interpreter Core messages: 340951 nosy: brett.cannon, eric.snow, steve.dower priority: normal severity: normal stage: needs patch status: open title: Warnings operate out of global runtime state. type: behavior versions: Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue36737> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36710] Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue36710> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36666] threading.Thread should have way to catch an exception thrown within
Eric Snow added the comment: Here's a basic decorator along those lines, similar to one that I've used on occasion: def as_thread(target): def _target(): try: t.result = target() except Exception as exc: t.failure = exc t = threading.Thread(target=_target) return t Sure, it's border-line non-trivial, but I'd hardly call it "exceptionally complicated". Variations for more flexibility: def as_thread(target=None, **tkwds): # A decorator to create a one-off thread from a function. if target is None: # Used as a decorator factory return lambda target: as_thread(target, **tkwds) def _target(*args, **kwargs): try: t.result = target(*args, **kwargs) except Exception as exc: t.failure = exc t = threading.Thread(target=_target, **tkwds) return t def threaded(target, **tkwds): # A decorator to produce a started thread when the "function" is called. if target is None: # Used as a decorator factory return lambda target: as_thread(target, **tkwds) @functools.wraps(target) def wrapper(*targs, **tkwargs) def _target(*args, *kwargs): try: t.result = target(*args, **kwargs) except Exception as exc: t.failure = exc t = threading.Thread(target=_target, args=targs, kwargs=tkwargs, **tkwds) t.start() return t return wrapper -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue3> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31954] Don't prevent dict optimization by coupling with OrderedDict
Eric Snow added the comment: Please don't miss the fact that the main reason for mirroring the dict table is to get O(1) node lookup (in the linked list). Otherwise most lookup-dependent operations, like __delitem__(), would become O(n); whereas in the pure-Python implementation they are O(1). This is all explained in the notes at the top of Objects/odictobject.c. Also, I didn't change anything in the dict implementation to rely on the OrderedDict implementation. So while I would say OrderedDict is coupled to dict, I wouldn't say the reverse, that dict is coupled to OrderedDict. If dict changes then OrderedDict must be updated apporpropriately, but not vice-versa. That should still hold. -- ___ Python tracker <https://bugs.python.org/issue31954> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: @Victor, I set up a FreeBSD 12.0 VM (in Hyper-v) and made sure core files were getting generated for segfaults. Then I cloned the cpython repo, built it (using GCC), and ran regrtest as you recommended. It generated no core files after half an hour. I adjusted the VM down to 1 CPU from 4 and there were no segfaults over an hour and a half of running those 4 test loops. So I've set the VM to 10% of a CPU and still have gotten no core files after over half an hour. The load average has been hovering between 5 and 6. I guess I'm not starving the VM enough. :) Any ideas of how far I need to throttle the VM? Is there more than just CPU that I need to limit? -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: New changeset b75b1a3504a0cea6fac6ecba44c10b2629577025 by Eric Snow in branch 'master': bpo-33608: Revert "Factor out a private, per-interpreter _Py_AddPendingCall()." (gh-12806) https://github.com/python/cpython/commit/b75b1a3504a0cea6fac6ecba44c10b2629577025 -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Change by Eric Snow : -- pull_requests: +12733 ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: Thanks for checking, Victor. Don't feel bad about your results, nor about not checking sooner. :) We'll get this sorted out. For now I'll revert. This is not code that changes very often, so there isn't much benefit to keeping it merged. Testing against a separate branch is just as easy. Could you point me at an immage for that VM or instructions on how to reproduce it? I hate having to bother you to test my changes! :) -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: New changeset f13c5c8b9401a9dc19e95d8b420ee100ac022208 by Eric Snow in branch 'master': bpo-33608: Factor out a private, per-interpreter _Py_AddPendingCall(). (gh-12360) https://github.com/python/cpython/commit/f13c5c8b9401a9dc19e95d8b420ee100ac022208 -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19476] Add a dedicated specification for module "reloading" to the language reference
Eric Snow added the comment: Thanks for checking in, Cheryl! Clearly no one picked up this banner. :) Furthermore, I'm not aware of any reload-related complaints coming from the community. I know of only a couple major use cases for reload(): refresh parts of a running app during development and reload a config (e.g. SIGHUP). Those use cases seem to be mostly stable. Given ~4.5 years of zero activity here, I think it's safe to say nothing further is going to happen. :) So I'm closing this issue. At this point I don't see much value in keeping it open. If someone later finds a concrete motivator for more detail about module-reloading in the language reference (or decide they want to drive this effort) then they can re-open this issue (or create a new one) at that point. -- resolution: -> rejected stage: needs patch -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue19476> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32280] Expose `_PyRuntime` through a section name
Change by Eric Snow : -- versions: +Python 3.8 -Python 3.7 ___ Python tracker <https://bugs.python.org/issue32280> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36487] Make C-API docs clear about what the "main" interpreter is
Eric Snow added the comment: I should have added something like this earlier, but here are key points to consider covering: * "main" interpreter is the original, created when the runtime initializes * historically almost always the only Python interpreter in a process * this is changing (more projects using subinterpreters, PEP 554) * in the "python" executable, subinterpreters available via extension modules (but uncommon) * it has extra responsiblities: * signal handling, in main thread * "pending calls", in main thread * not necessarily something to publicize :) * running in the "main" thread * can be taken over by a sub-interpreter, but not likely * execution *during* runtime initialization * usually (always?) the active interpreter during runtime finalization * others? * no strong link between interpreters (e.g. main <--> sub <--> sub) * no record of what interpreter was active (if any) when a subinterpreter was created * no "parent" interpreter * thread states may share thread ID * a bunch of stuff in the C-API and in the runtime still assumes that the main interpreter is running in the current OS thread * other stuff? Not all of that necessarily belongs there in the docs, but a lot of it does. :) -- nosy: +ncoghlan ___ Python tracker <https://bugs.python.org/issue36487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36157] Document PyInterpreterState_Main().
Eric Snow added the comment: Thanks for working on this, Joannah! :) -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36157> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36487] Make C-API docs clear about what the "main" interpreter is
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue36487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33356] Windows 10 buildbot: test__xxsubinterpreters.test_already_running() fails randomly
Eric Snow added the comment: I'll take a look when I get a chance. -- ___ Python tracker <https://bugs.python.org/issue33356> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36427] Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread
Eric Snow added the comment: > Currently PyEval_RestoreThread and its callers (mainly PyGILState_Ensure) > can terminate the thread if the interpreter is finalizing: s/interpreter/runtime/ Most likely (guaranteed?) it will be in the main interpreter, but it is actually not triggered by interpreter finalization (though it probably should be; I've opened #36479). -- ___ Python tracker <https://bugs.python.org/issue36427> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36479] Exit threads when interpreter is finalizing rather than runtime.
New submission from Eric Snow :
Currently when a thread acquires the GIL, it subsequently exits if the runtime
is finalizing. This helps with some cases like with stopping daemon threads.
This behavior should instead be triggered by the thread's interpreter
finalizing rather than the runtime. This implies the following changes:
* in Py_EndInterpreter() move "interp-finalizing = 1;" to right after the call
to "wait_for_thread_shutdown()"
* in Py_FinalizeEx() add "interp-finalizing = 1;" right after the call to
"wait_for_thread_shutdown()"
* update _PyEval_EvalFrameDefault() (the eval loop) to check
"interp->finalizing" instead of the runtime
* likewise update PyEval_RestoreThread() (as well as PyEval_AcquireThread() and
PyEval_AcquireLock(); see #36475)
The check will actually need to change from this:
if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate)) {
to look like this:
if (interp->finalizing && !_Py_CURRENTLY_FINALIZING(tstate)) {
If we could guarantee that only the main thread will ever call Py_FinalizeEx()
then it would look more like this:
if (interp->finalizing && tstate.id != _PyRuntime.main_thread {
--
components: Interpreter Core
messages: 339155
nosy: eric.snow
priority: normal
severity: normal
stage: needs patch
status: open
title: Exit threads when interpreter is finalizing rather than runtime.
type: behavior
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue36479>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36427] Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread
Eric Snow added the comment: > > if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate)) > > _Py_IsFinalizing() check is redundant :-) Not really: * _Py_IsFinalizing() checks if the runtime is finalizing * _Py_CURRENTLY_FINALIZING checks if the current thread is the one running finalization -- ___ Python tracker <https://bugs.python.org/issue36427> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36427] Document that PyEval_RestoreThread and PyGILState_Ensure can terminate the calling thread
Eric Snow added the comment: Related: * #36475: "PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly." * #36476: "Runtime finalization assumes all other threads have exited." -- ___ Python tracker <https://bugs.python.org/issue36427> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36157] Document PyInterpreterState_Main().
Eric Snow added the comment: As I noted on the PR, this might be a good chance to make sure the C-API docs are clear about what the "main" interpreter is. -- ___ Python tracker <https://bugs.python.org/issue36157> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36097] Use only public C-API in _xxsubinterpreters module.
Change by Eric Snow : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36097> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36225] Lingering subinterpreters should be implicitly cleared on shutdown
Eric Snow added the comment: test__xxsubinterpreters is a great place for tests that exercise use of subinterpreters, including most lifecycle operations. There are also one or two subinterpreter-related tests in test_embed. However, for this issue the interplay with runtime finalization means tests should probably stay with other tests that exercise Py_FinalizeEx(). -- ___ Python tracker <https://bugs.python.org/issue36225> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36225] Lingering subinterpreters should be implicitly cleared on shutdown
Eric Snow added the comment: Interestingly, I noticed this independently today. :) Here's what I wrote in #36477 (which I've closed as a duplicate): When using subinterpreters, any that exist when Py_FinalizeEx() is called do not appear to get cleaned up during runtime finalization. Maybe I've been looking at the code too much and I'm missing something. :) This really isn't a problem except for embedders that use subinterpreters (where we're leaking memory). However, even with the "python" executable it can have an impact because the subinterpreters' non-daemon threads will exit later than expected. (see #36469 & #36476) The solution would be to finalize all subinterpreters at the beginning of Py_FinalizeEx(), right before the call to wait_for_thread_shutdown(). This means calling Py_EndInterpreter() for all the runtime's interpreters (except the main one). It would also mean setting a flag (_PyRuntime.interpreters.finalizing?) right before that to disallow creation of any more subinterptreters. -- ___ Python tracker <https://bugs.python.org/issue36225> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36477] Subinterpreters are not finalized during runtime finalization.
Change by Eric Snow : -- resolution: -> duplicate stage: needs patch -> resolved status: open -> closed superseder: -> Lingering subinterpreters should be implicitly cleared on shutdown ___ Python tracker <https://bugs.python.org/issue36477> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: @Remy, aside from the recommendations I've made, I'm not sure what else we can do to help. Before we close the issue, I'd really like to ensure that one of those threads is holding the GIL still. It would definitely be a problem if a thread exited while still holding the GIL. The only way I can think of is to trace through the code. [1] [1] https://github.com/python/cpython/tree/v3.5.3 -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36476] Runtime finalization assumes all other threads have exited.
Eric Snow added the comment: FYI, I've opened issue36477 to deal with the subinterpreters case. -- ___ Python tracker <https://bugs.python.org/issue36476> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36477] Subinterpreters are not finalized during runtime finalization.
New submission from Eric Snow : When using subinterpreters, any that exist when Py_FinalizeEx() is called do not appear to get cleaned up during runtime finalization. Maybe I've been looking at the code too much and I'm missing something. :) This really isn't a problem except for embedders that use subinterpreters (where we're leaking memory). However, even with the "python" executable it can have an impact because the subinterpreters' non-daemon threads will exit later than expected. (see #36469 & #36476) The solution would be to finalize all subinterpreters at the beginning of Py_FinalizeEx(), right before the call to wait_for_thread_shutdown(). This means calling Py_EndInterpreter() for all the runtime's interpreters (except the main one). It would also mean setting a flag (_PyRuntime.interpreters.finalizing?) right before that to disallow creation of any more subinterptreters. -- components: Interpreter Core messages: 339144 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Subinterpreters are not finalized during runtime finalization. type: behavior versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue36477> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: I've also opened issues #36476 and #36477. -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36476] Runtime finalization assumes all other threads have exited.
New submission from Eric Snow : Among the first 3 things that happen in Py_FinalizeEx() are, in order: 1. wait for all non-daemon threads (of the main interpreter) to finish 2. call registered atexit funcs 3. mark the runtime as finalizing At that point the only remaining Python threads are: * the main thread (where finalization is happening) * daemon threads * non-daemon threads created in atexit functions * any threads belonging to subinterpreters The next time any of those threads (aside from main) acquire the GIL, we expect that they will exit via a call to PyThread_exit_thread() (caveat: issue #36475). However, we have no guarantee on when that will happen, if ever. Such lingering threads can cause problems, including crashes and deadlock (see issue #36469). I don't know what else we can do, beyond what we're already doing. Any ideas? -- components: Interpreter Core messages: 339143 nosy: eric.snow priority: normal severity: normal status: open title: Runtime finalization assumes all other threads have exited. type: behavior versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue36476> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36475] PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly.
Change by Eric Snow : -- components: +Interpreter Core ___ Python tracker <https://bugs.python.org/issue36475> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: Looking at the stack traces for all your threads (super helpful, BTW), I saw 4 groups: * (1) main thread (blocked on GIL during runtime finalization) + Thread 1 * (12) blocked on GIL in call to PyEval_RestoreThread() (socket, select, time.sleep(), file.read()) * (3) waiting for a connection on a socket (with GIL not held, presumably) + Thread 5 + Thread 12 + Thread 14 * (1) blocked on a threading.Lock + Thread 7 * (1) blocked on "head" lock when cleaning up after execution + Thread 18 So there's a third lock involved in this deadlock. It isn't actually clear to me (without further digging) which thread actually holds the GIL. I'd guess it's one of the last two (though it could be one of the 3 waiting on a socket). However, in each of those cases I would not expect the GIL to be held at that point. Regardless, the race likely involves the threading.Lock being held late in finalization. It could be that you are not releasing it somewhere where you should be. -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: I've opened issue36475 for the two C-API functions that do not cause daemon threads to exit. -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36475] PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly.
New submission from Eric Snow : Daemon threads keep running until they finish or until finalization starts. For the latter, there is a check right after the thread acquires the GIL which causes the thread to exit if runtime finalization has started. [1] However, there are functions in the C-API that facilitate acquiring the GIL, but do not cause the thread to exit during finalization: PyEval_AcquireLock() PyEval_AcquireThread() Daemon threads that acquire the GIL through these can cause a deadlock during finalization. (See issue #36469.) They should probably be updated to match what PyEval_RestoreThread() does. [1] see PyEval_RestoreThread() and the eval loop, in PyEval_EvalFrameEx() -- messages: 339138 nosy: eric.snow priority: normal severity: normal stage: test needed status: open title: PyEval_AcquireLock() and PyEval_AcquireThread() do not handle runtime finalization properly. type: behavior versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue36475> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: Here are some things that would likely help: * in the main thread stop your daemon threads if you hit SystemExit * make sure daemon threads release/acquire the GIL frequently (so they notice finalization) * make sure daemon threads otherwise exit promptly (no long running C code) * stop using daemon threads * use a newer version of Python (may be fixed there) I'm going take a look at master to see if it has a similar possible problem with daemon threads and runtime finalization. If there is then I'll likely open a separate issue (and reference it here). -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: At this point I think it's likely that the problem relates to how daemon threads are handled during runtime finalization. What normally happens in the main thread of the "python3" executable is this: 1. Python runtime initializes 2. main interpreter initializes 3. the requested code is executed (in this case via PyRun_SimpleFileExFlags) 4. the runtime is finalized a. wait for all non-daemon threads to finish b. call registered atexit funcs c. mark the runtime as finalizing d. ... >From the stack trace you gave, the main thread is definitely past step 4c in >the runtime finalization process. Note the following: * marking the runtime as finalizing is meant to cause all remaining daemon threads to exit the next time they take the GIL * further, runtime finalization assumes that all daemon threads will have finished soon after step 4c * not all C-API functions for acquiring the GIL actually cause the current thread to exit if the runtime is finalizing (see below) * step 4a applies only to the main interpreter; using subinterpreters complicates the situation a little (threads get finalized with each subinterpreter later on) * any thread created in an atexit func (4b) can cause problems Cause thread to exit if runtime is finalizing: * PyEval_RestoreThread() * PyEval_EvalFrameEx() (the eval loop) Do not cause thread to exit if runtime is finalizing: * PyEval_InitThreads() * PyEval_ReInitThreads() * PyEval_AcquireLock() * PyEval_AcquireThread() Regardless, from what you've reported it looks like the following is happening: m1. main thread starts m2. thread A (daemon) created m3. thread B (daemon) created (by main thread or thread A) m4. code in main thread raises SystemExit m5. the Python runtime begins finalization (incl. steps 4a-4c above) m6. the main thread starts cleaning up the main interpreter m7. it starts cleaning up the main interpreter's threads m8. it acquires the "head" lock m9. it starts cleaning up the frame tied to one of those threads m10. a socket object in that frame gets destroyed m11. a warning is issued (presumably about an unclosed socket) tB1. thread B (still running) acquires GIL tB2. it does not release the GIL m12. creating the warning causes a function to get called (socket.__repr__) m13. this causes the main thread to try to acquire the GIL m14. it blocks (waiting for thread B) tA1. thread A (still running) finishes and starts cleaning itself up tA2. it tries to acquire the "head" lock tA3. it blocks (waiting for the main thread) Notable: * at step m7 the code assumes that all the interpreter's threads have exited at that point * with the verbose flag to "python3", the runtime will print a warning if any thread is still running when its (finalizing) interpreter tries to clean it up -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: * Are any signals (or signal handlers) involved? * Are you monkeypatching any builtins, builtin/stdlib modules (or any parts of those)? Keep in mind that a number of bugs have been fixed in later releases related to the various things I've asked about. For instance, see issue #30703. -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30703] Non-reentrant signal handler (test_multiprocessing_forkserver hangs)
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue30703> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: Also: * are any of your threads daemon threads? -- ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36469] Stuck during interpreter exit, attempting to take the GIL
Eric Snow added the comment: As Inada-san indicated, the problem might be resolved already. So without the option of reproducing the problem, it will be hard to resolve this. Here's some information you could provide that might help narrow down the scope a bit: * what (stdlib/third-party/internal) modules are you using, particularly extension modules? * how many threads are you using at once? * how many atexit handlers do you have and what are they each doing? -- nosy: +eric.snow type: -> behavior ___ Python tracker <https://bugs.python.org/issue36469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36450] 3D Array Assignment to all 2D sub-arrays
Eric Snow added the comment: In Python, multiplication on a list does not make copies of the values in the original list. So what you have done is equivalent to the following: a = [0, 0] b = [a, a] M = [b, b] Hence: >>> M[0] is M[1] True >>> M[0][0] is M[0][1] True >>> M[1][0] is M[1][1] True >>> M[0][0] is M[1][0] True So the following *all* modify the first value in "a": M[0][0][0] = 1 M[1][0][0] = 1 M[0][1][0] = 1 M[1][1][0] = 1 That is why you are seeing the result you reported. Depending on your needs, better solutions include using a list comprehension, spelling out the for loops (for readability), a helper function (for complex problems), or simply spelling out the full list literal instead of composing it. Regardless, I highly recommend using a solution that is easy for a new reader to understand. Even if no one else will read this code, your six-months-from-now self will thank you. :) -- nosy: +eric.snow resolution: -> not a bug stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36450> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32312] Create Py_AtExitRegister C API
Eric Snow added the comment: > Neil Schemenauer added the comment: > Regarding m_traverse, maybe the list of finalizers should be stored somewhere > in the interpreter > state, not in the atexit module state. That would make more sense to me. > atexit could merely be > a way to manage it rather than actually holding it. +1 -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue32312> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36124] Provide convenient C API for storing per-interpreter state
Eric Snow added the comment: @arigo, thanks for nudging us here. :) Let me know if there's anything else that would help here. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36124> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36124] Provide convenient C API for storing per-interpreter state
Eric Snow added the comment: New changeset d2fdd1fedf6b9dc785cf5025b548a989faed089a by Eric Snow in branch 'master': bpo-36124: Add PyInterpreterState.dict. (gh-12132) https://github.com/python/cpython/commit/d2fdd1fedf6b9dc785cf5025b548a989faed089a -- ___ Python tracker <https://bugs.python.org/issue36124> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36097] Use only public C-API in _xxsubinterpreters module.
Eric Snow added the comment: New changeset c11183cdcff6af13c4339fdcce84ab63f7930ddc by Eric Snow in branch 'master': bpo-36097: Use only public C-API in the_xxsubinterpreters module (adding as necessary). (gh-12359) https://github.com/python/cpython/commit/c11183cdcff6af13c4339fdcce84ab63f7930ddc -- ___ Python tracker <https://bugs.python.org/issue36097> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Change by Eric Snow : -- pull_requests: +12327 ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36097] Use only public C-API in _xxsubinterpreters module.
Change by Eric Snow : -- pull_requests: +12326 stage: needs patch -> patch review ___ Python tracker <https://bugs.python.org/issue36097> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: New changeset 842a2f07f2f08a935ef470bfdaeef40f87490cfc by Eric Snow in branch 'master': bpo-33608: Deal with pending calls relative to runtime shutdown. (gh-12246) https://github.com/python/cpython/commit/842a2f07f2f08a935ef470bfdaeef40f87490cfc -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: New changeset 8479a3426eb7d1840473f7788e639954363ed37e by Eric Snow in branch 'master': bpo-33608: Make sure locks in the runtime are properly re-created. (gh-12245) https://github.com/python/cpython/commit/8479a3426eb7d1840473f7788e639954363ed37e -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Eric Snow added the comment: New changeset 5be45a6105d656c551adeee7770afdc3b806fbb5 by Eric Snow in branch 'master': bpo-33608: Minor cleanup related to pending calls. (gh-12247) https://github.com/python/cpython/commit/5be45a6105d656c551adeee7770afdc3b806fbb5 -- ___ Python tracker <https://bugs.python.org/issue33608> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
