[issue47185] code.replace(co_code=new_code) no longer catch exceptions on Python 3.11
Change by Eric Snow : -- nosy: +Mark.Shannon ___ Python tracker <https://bugs.python.org/issue47185> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: Looks like gh-32218 worked. -- status: open -> closed ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: Specifically: https://github.com/python/cpython/actions/workflows/build.yml. -- status: pending -> open ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: I'll keep an eye on PRs for the next day or so. -- resolution: -> fixed stage: patch review -> resolved status: open -> pending ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: New changeset e7bb7c2f047b4f97e4426c42ae209c969808069d by Eric Snow in branch 'main': bpo-47146: Stop Depending On regen-deepfreeze For regen-global-objects (gh-32218) https://github.com/python/cpython/commit/e7bb7c2f047b4f97e4426c42ae209c969808069d -- ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Change by Eric Snow : -- pull_requests: +30294 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/32218 ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: Brandt pointed out this is consistently reproducible locally: make clean regen-all -j I'll get this sorted out today. -- ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: I re-ran jobs that had failed before I merged that gh-32206. Several passed, but the following are still failing: * https://github.com/python/cpython/pull/32188 + https://github.com/python/cpython/runs/5773938424 * https://github.com/python/cpython/pull/32132 + https://github.com/python/cpython/runs/5774054192 * https://github.com/python/cpython/pull/32177 + https://github.com/python/cpython/runs/5773949869 Back to the drawing board... -- stage: patch review -> needs patch ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: New changeset db4dada5108dd49ebca23e4559a53630a2df8447 by Eric Snow in branch 'main': bpo-47146: Avoid Using make Recursively (gh-32206) https://github.com/python/cpython/commit/db4dada5108dd49ebca23e4559a53630a2df8447 -- ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Change by Eric Snow : -- pull_requests: +30281 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/32206 ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: Looks like this is still an intermittent problem: * https://github.com/python/cpython/pull/32195 + failed: https://github.com/python/cpython/runs/5756616733 + failed: https://github.com/python/cpython/runs/5753267869 + failed: https://github.com/python/cpython/runs/5757169625 * https://github.com/python/cpython/pull/32114 + failed: https://github.com/python/cpython/runs/5756616733 + passed: https://github.com/python/cpython/runs/5757213346 * https://github.com/python/cpython/pull/32186 + failed: ... + passed: https://github.com/python/cpython/runs/5757178754 Per Mark Shannon (on discord): The "Check if generated files are up to date" is still failing consistently. It looks like the makefile is missing a dependency on the ./Programs/_freeze_module for targets that require /Programs/_freeze_module -- resolution: fixed -> stage: resolved -> needs patch status: closed -> open ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: Looks like that fixed it, per https://github.com/python/cpython/pull/32134. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: New changeset 4c116f716bd1c174d6530b9a7a5ed3863927a109 by Eric Snow in branch 'main': bpo-47146: Eliminate a race between make regen-deepfreeze and make regen-global-objects. (gh-32162) https://github.com/python/cpython/commit/4c116f716bd1c174d6530b9a7a5ed3863927a109 -- ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Change by Eric Snow : -- keywords: +patch pull_requests: +30240 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/32162 ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
Eric Snow added the comment: There's probably something racy with make. See: https://github.com/python/cpython/runs/5712538599?check_suite_focus=true#step:10:1147 -- ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47146] PR check "Check if generated files are up to date" failing intermittently
New submission from Eric Snow : The "Check if generated files are up to date" GitHub check for PRs has been failing recently. It may also impact local usage of "make regen-all". Example: https://github.com/python/cpython/runs/5719012664 This may be related to gh-32061. -- assignee: eric.snow components: Build messages: 416193 nosy: Mark.Shannon, brandtbucher, eric.snow priority: normal severity: normal stage: needs patch status: open title: PR check "Check if generated files are up to date" failing intermittently versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue47146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46712] Share global string identifiers in deepfreeze
Eric Snow added the comment: New changeset febf54bcf3fdc45ad84b4073e24bbaaee0ac8b2a by Eric Snow in branch 'main': bpo-46712: Do not Regen Deep-Frozen Modules before Generating Global Objects (gh-32061) https://github.com/python/cpython/commit/febf54bcf3fdc45ad84b4073e24bbaaee0ac8b2a -- ___ Python tracker <https://bugs.python.org/issue46712> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 21412d037b07c08266e96dfd0c0e44a1b7693bc1 by Eric Snow in branch 'main': bpo-46541: Add a Comment About When to Use _Py_DECLARE_STR(). (gh-32063) https://github.com/python/cpython/commit/21412d037b07c08266e96dfd0c0e44a1b7693bc1 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +30154 pull_request: https://github.com/python/cpython/pull/32063 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46964] The global config should not be stored on each interpreter
Eric Snow added the comment: Here are reasons why PyConfig relates to the runtime and not each interpreter: * PyConfig was introduced so embedders could dictate how the *runtime* should be initialized * after initialization, it represents how the runtime was initialized * in the public API, PyConfig relates only to the runtime * more than a few PyConfig fields are specific to the entire runtime and not appropriate on a per-interpreter basis * such PyConfig fields are only used during runtime init * currently, every interpreter uses (a copy of) the original PyConfig, completely unchanged * there is no API for creating an interpreter with a different config * most of the things that one might want to customize on an interpreter can already be done right after the interpreter is initialized * thus far we have had no actual use cases for initializing an interpreter with a different config * for many of the PyConfig fields, allowing different values for each interpreter is an attractive nuisance and would invite confusion This is why PyConfig makes more sense as the global config, not a per-interpreter one. I think it was a mistake to store the config on PyInterpreterState. Keep in mind, though, that PyConfig was added several years before _PyRuntimeState existed. If _PyRuntimeState had been around, I'm sure that's where we would have kept the global config. Thus, it makes sense to move PyInterpreterState.config to _PyRuntimeState.config. If there's a need for a per-interpreter config later, we would do the following: * add a new PyInterpreterConfig with only the fields we need * add a new PyInterpreterState.config field of that type In fact, that is exactly what I'm proposing in PEP 684, where there is a need for creating an interpreter with various isolation options. That's what got me thinking about why PyConfig isn't good for customizing new interpreters. Even if we didn't move PyInterpreterState.config to _PyRuntimeState.config, PEP 684 would still not use PyConfig as the config to pass when creating a new interpreter. Using PyConfig would be confusing and an invitation for trouble. So I'd use a new PyInterpreterConfig either way. Consequently, having PyConfig on PyInterpreterState would be confusing, with no benefit. My intention with this BPO issue was to get our internal details aligned with what makes sense for a global config and set the stage for adding a proper per-interpreter config. -- nosy: +ncoghlan ___ Python tracker <https://bugs.python.org/issue46964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46712] Share global string identifiers in deepfreeze
Change by Eric Snow : -- pull_requests: +30151 pull_request: https://github.com/python/cpython/pull/32061 ___ Python tracker <https://bugs.python.org/issue46712> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46712] Share global string identifiers in deepfreeze
Eric Snow added the comment: > After a new `&_Py_ID(__orig_class__)` is added to > Objects/genericaliasobject.c, running `make regen-global-objects` starts > > gcc -pthread -c [snipped] -DPy_BUILD_CORE -o Objects/genericaliasobject.o > Objects/genericaliasobject.c > > which fails with a compilation error because that identifier is not yet > defined. Is there a good way to convince `make` to regenerate the global > objects without this sort of circular dependency? Am I missing a step? I'm looking into this. A temporary workaround is to run Tools/scripts/generate-global-objects.py directly. -- ___ Python tracker <https://bugs.python.org/issue46712> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46964] The global config should not be stored on each interpreter
Change by Eric Snow : -- keywords: +patch pull_requests: +29879 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31771 ___ Python tracker <https://bugs.python.org/issue46964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46964] The global config should not be stored on each interpreter
New submission from Eric Snow : tl;dr let's move PyInterpreterState.config to _PyRuntimeState.config. Historically the runtime has been initialized using Py_Initialize(). PEP 587 added Py_InitializeFromConfig(), which takes a PyConfig and allows all sorts of customization of the runtime. This is valuable for embedders (and benefits core development too). During runtime initialization the given config is copied and stored internally on the main interpreter. Once initialization completes, the config is no longer modified. The config values are then used in a variety of places during execution. If a new interpreter is created then the config from the current (or main) interpreter are copied into it. Note the following: * the config is never modified * there is no public API for getting the config or changing it * there is no API for creating an interpreter with a different config * the fact that the config is stored on the interpreter is an internal detail and not documented (nor discussed in PEP 587) Consequently, PyConfig really is the global runtime config. Yet we are storing a copy of it on each interpreter. Doing so unnecessarily adds extra complexity (and, when multiple interpreters are used, extra CPU usage and extra memory usage). So I propose that we move the config to _PyRuntimeState. The change isn't big nor all that complex. Note that actually there is one field that can differ between interpreters: PyConfig._isolated_interpreter (set in _Py_NewInterpreter()). We can move that one field to a new per-interpreter config struct. -- assignee: eric.snow components: C API, Interpreter Core messages: 414772 nosy: eric.snow, vstinner priority: normal severity: normal stage: needs patch status: open title: The global config should not be stored on each interpreter versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46712] Share global string identifiers in deepfreeze
Eric Snow added the comment: New changeset 21099fc064c61d59c936a2f6a0db3e07cd5c8de5 by Eric Snow in branch 'main': bpo-46712: Let generate_global_objects.py Run on Earlier Python Versions (gh-31637) https://github.com/python/cpython/commit/21099fc064c61d59c936a2f6a0db3e07cd5c8de5 -- ___ Python tracker <https://bugs.python.org/issue46712> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46890] venv does not create "python" link in python 3.11
Eric Snow added the comment: This may be related to the getpath.py work Steve did. -- nosy: +eric.snow, steve.dower ___ Python tracker <https://bugs.python.org/issue46890> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46712] Share global string identifiers in deepfreeze
Change by Eric Snow : -- pull_requests: +29759 pull_request: https://github.com/python/cpython/pull/31637 ___ Python tracker <https://bugs.python.org/issue46712> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46753] Statically allocate and initialize the empty tuple.
Change by Eric Snow : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue46753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46886] pyexpat occasionally fails to build on the ARM64 Windows Non-Debug 3.x buildbot
New submission from Eric Snow : example: https://buildbot.python.org/all/#/builders/730/builds/4081 -- components: Build messages: 414223 nosy: eric.snow, vstinner priority: normal severity: normal stage: needs patch status: open title: pyexpat occasionally fails to build on the ARM64 Windows Non-Debug 3.x buildbot versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46886> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46753] Statically allocate and initialize the empty tuple.
Eric Snow added the comment: New changeset 08deed1af56bec8668c6cb4d5cfd89e393e1fe5e by Eric Snow in branch 'main': bpo-46753: Add the empty tuple to the _PyRuntimeState.global_objects. (gh-31345) https://github.com/python/cpython/commit/08deed1af56bec8668c6cb4d5cfd89e393e1fe5e -- ___ Python tracker <https://bugs.python.org/issue46753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45316] [C API] Functions not exported with PyAPI_FUNC()
Eric Snow added the comment: Thanks for working on this, Victor. -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue45316> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46806] Overlapping PYTHONPATH may cause import already imported module
Eric Snow added the comment: FYI, a technical solution has been discussed before: bpo-13475 and PEP 395. However, that does not help so much if the default behavior isn't changed. That would require a PEP but I expect it would be rejected because so many scripts already rely on the current behavior and the current behavior is useful in some cases. PEP 395 also has a good discussion of the various pitfalls related to sys.path[0] initialization. Furthermore, the topic is discussed in quite a few issues, such as bpo-44132 and bpo-29929. Probably the best use of your time on this would be to improve the documentation so people will more easily avoid the problem, or at least more easily diagnose the situation when they stumble on it. Again, PEP 395 is a good guide for this. -- ___ Python tracker <https://bugs.python.org/issue46806> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46765] Replace Locally Cached Strings with Statically Initialized Objects
Change by Eric Snow : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue46765> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46765] Replace Locally Cached Strings with Statically Initialized Objects
Eric Snow added the comment: New changeset 1f455361ecfb1892e375bdbee813cdf095b6cfb8 by Eric Snow in branch 'main': bpo-46765: Replace Locally Cached Strings with Statically Initialized Objects (gh-31366) https://github.com/python/cpython/commit/1f455361ecfb1892e375bdbee813cdf095b6cfb8 -- ___ Python tracker <https://bugs.python.org/issue46765> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting and immortal objects
Eric Snow added the comment: Thanks for the updates, Eddie. -- ___ Python tracker <https://bugs.python.org/issue40255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46806] Overlapping PYTHONPATH may cause import already imported module
Eric Snow added the comment: I'm leaving this "pending" in case there may be some improvement we can make to the documentation to address this. -- components: +Interpreter Core nosy: +docs@python status: open -> pending ___ Python tracker <https://bugs.python.org/issue46806> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46806] Overlapping PYTHONPATH may cause import already imported module
Eric Snow added the comment: Here is more detail on what happens when "from src import user_1, user_2, user_3" is executed in main.py: 1. the "src" module is imported a. not found in sys.modules b. file found on a sys.path entry (the directory main.py is in) c. the "src" module is created with __path__ set to the src directory d. the module is added to sys.modules e. src/__init__.py is executed 2. the "src.user_1" module is imported a. not found in sys.modules b. file found relative to src.__path__ c. module created d. added to sys.modules e. executed 3. "from .common_object import OBJECT" is resolved to "from src.common_object import OBJECT" 4. "src.common_object" is imported (see 2) 5. src.common_object.OBJECT is created 6. "src.user_2" is imported (see 2) 7. "src.common_object" is already found in sys.modules and used 8. "src.user_3" is imported (see 2) 9. "common_object" is imported a. not found in sys.modules b. file found on a sys.path entry (the one you added in main.py) c. module created d. added to sys.modules e. executed 10. common_object.OBJECT is created So the module created at (4) is different than the one at (9), even though they are imported from the same file. Consequently, the OBJECT in each is likewise distinct. -- ___ Python tracker <https://bugs.python.org/issue46806> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46806] Overlapping PYTHONPATH may cause import already imported module
Eric Snow added the comment: When you run a Python script, the directory the script is in is automatically added to the beginning of sys.path. This is the fundamental issue you've run into. Basically, "src.common_object" is imported relative to the "src" that was imported relative to that automatically added sys.path entry. However, "common_object" is a distinct module imported relative to the sys.path entry you explicitly added. In general, adding a package's directory to sys.path is a bad idea. -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue46806> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41111] [C API] Convert a few stdlib extensions to the limited C API (PEP 384)
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue4> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43974] Define Py_BUILD_CORE_MODULE in extensions instead of setup.py and Modules/Setup
Change by Eric Snow : -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue43974> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: (from https://github.com/python/cpython/pull/31376#issuecomment-1041836106) [corona10] > Should we create the separate bpo issue if module changes are too noisy? I think it's fine to use the one issue. There are only 26 modules with `NEEDS_PY_IDENTIFIER` and none have much risk with the change. So it's unlikely that we'll get any discussion about any specific module. If we do, we can easily create an issue then for the module in question. If one of the modules had many uses of `_Py_IDENTIFIER()` then it might make sense to split it out, but IIRC none of them have very many. -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46772] Statically Initialize PyArg_Parser in clinic.py
Change by Eric Snow : -- dependencies: +Add a Private API for Looking Up Global Objects, Statically allocate and initialize the empty tuple. ___ Python tracker <https://bugs.python.org/issue46772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46773] Add a Private API for Looking Up Global Objects
New submission from Eric Snow : We need this to statically initialize PyArg_Parser.kwtuple. (See bpo-46772.) For now this will be a "private" API (leading underscore). Ultimately, we'll want a Public API, so we can eventually stop exposing *any* objects as symbols in the C-API. However, that will need a PEP. -- assignee: eric.snow components: Interpreter Core messages: 413352 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Add a Private API for Looking Up Global Objects versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46773> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46772] Statically Initialize PyArg_Parser in clinic.py
New submission from Eric Snow : The code generated by clinic.py is already partially statically initialized. Currently we init the other fields in Python/getargs.c:parser_init(), which runs the first time we try to use each parser. AFAICS, that remaining init that could be done statically using the data we have available in clinic.py during code generation. My primary interest is in static init of PyArg_Parser.kwtuple, which is a tuple containing only strings. -- assignee: eric.snow components: Interpreter Core messages: 413351 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Statically Initialize PyArg_Parser in clinic.py versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46772> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46765] Replace Locally Cached Strings with Statically Initialized Objects
Change by Eric Snow : -- keywords: +patch pull_requests: +29516 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31366 ___ Python tracker <https://bugs.python.org/issue46765> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46765] Replace Locally Cached Strings with Statically Initialized Objects
New submission from Eric Snow : This removes a number of static variables and is a little more efficient. -- assignee: eric.snow components: Interpreter Core messages: 413313 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Replace Locally Cached Strings with Statically Initialized Objects versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46765> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 4d8a515d193a4c9f3844704f974ddb870d7ee383 by Eric Snow in branch 'main': bpo-46541: Scan Fewer Files in generate_global_objects.py (gh-31364) https://github.com/python/cpython/commit/4d8a515d193a4c9f3844704f974ddb870d7ee383 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 6c8958948666403f2370ca7b4c0a52b2010ec16d by Eric Snow in branch 'main': bpo-46541: Drop the check for orphaned global strings. (gh-31363) https://github.com/python/cpython/commit/6c8958948666403f2370ca7b4c0a52b2010ec16d -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +29514 pull_request: https://github.com/python/cpython/pull/31364 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +29513 pull_request: https://github.com/python/cpython/pull/31363 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 12360aa159c42c7798fd14225d271e6fd84db7eb by Eric Snow in branch 'main': bpo-46541: Discover the global strings. (gh-31346) https://github.com/python/cpython/commit/12360aa159c42c7798fd14225d271e6fd84db7eb -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +29496 pull_request: https://github.com/python/cpython/pull/31346 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46753] Statically allocate and initialize the empty tuple.
Change by Eric Snow : -- keywords: +patch pull_requests: +29495 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/31345 ___ Python tracker <https://bugs.python.org/issue46753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46753] Statically allocate and initialize the empty tuple.
Eric Snow added the comment: Also see bpo-45953. -- ___ Python tracker <https://bugs.python.org/issue46753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46753] Statically allocate and initialize the empty tuple.
New submission from Eric Snow : Currently it is created dynamically from the tuple freelist. -- assignee: eric.snow components: Interpreter Core messages: 413268 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Statically allocate and initialize the empty tuple. versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46753> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- pull_requests: +29494 pull_request: https://github.com/python/cpython/pull/31344 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: With core code sorted out, stdlib and 3rd party extension modules are left to sort out. I see the following possibilities: 1. leave `_Py_IDENTIFIER()` alone (it is already disallowed in core code) 2. change `_Py_IDENTIFIER()` to create static string objects (then get rid of global state) 3. get rid of `_Py_IDENTIFIER()` a. provide a public alternative (needs a PEP) b. first work with 3rd party projects to stop using it -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- Removed message: https://bugs.python.org/msg413241 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: With core code sorted out, stdlib and 3rd party extension modules are left to sort out. I see the following possibilities: * leave `_Py_IDENTIFIER()` alone (it is already disallowed in core code) * change `_Py_IDENTIFIER()` to create static string objects (then get rid of global state) * get rid of `_Py_IDENTIFIER()` a. provide a public alternative (needs a PEP) b. first work with 3rd party projects to stop using it -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46748] Python.h includes stdbool.h
Eric Snow added the comment: On Mon, Feb 14, 2022 at 2:28 AM Petr Viktorin wrote: > Eric, is this necessary? Would an old-school `int` do? > Or should we say it's 2022 already and everyone needs to use stdbool.hfore > bools? I started using ``bool`` (stdbool.h) when I saw it specified in PEP 7 (https://www.python.org/dev/peps/pep-0007/#c-dialect). If it is a problem then I think it should be answered relative to PEP 7. I'm not sure what the best route is though. Personally, I'd argue that if it's in C99 then it should probably be standard at this point. :) However, I'm not opposed to going back to plain int. (And I know there are definitely a number of old-school folks who prefer plain int. :) ) -- ___ Python tracker <https://bugs.python.org/issue46748> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: On Fri, Feb 11, 2022 at 1:36 AM Christoph Reiter wrote: > Sorry if off topic, but I noticed that CPython doesn't deprecate macros in > code, while with gcc/clang it's possible to show compiler warnings for them > using some pragma magic: > [snip] > Maybe that makes getting rid of them easier in the long run? That's a good question. We do have Py_DEPRECATED() (in Include/pyport.h), which is used for symbols. I'm not sure anyone has given much thought to deprecating macros, but it's probably worth considering. I recommend that you post something about this to python-...@python.org. FWIW, here are other explanations of how to deprecate macros: * https://stackoverflow.com/questions/57478368/what-is-the-best-way-to-mark-macro-as-deprecated/57479189#57479189 * https://stackoverflow.com/questions/2681259/how-to-deprecate-a-c-pre-processor-macro-in-gcc/29297970#29297970 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Eric Snow added the comment: New changeset 80e4f262aa27a39abf3fadc19a6323fea4607a8f by Eric Snow in branch 'main': bpo-36876: Make sure the c-analyzer is checking all the source files.' (gh-31264) https://github.com/python/cpython/commit/80e4f262aa27a39abf3fadc19a6323fea4607a8f -- ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Change by Eric Snow : -- pull_requests: +29430 pull_request: https://github.com/python/cpython/pull/31264 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Eric Snow added the comment: New changeset cb68788dcadf43b47292bab7816a5ed9efa69730 by Eric Snow in branch 'main': bpo-36876: Minor cleanup to c-analyzer "ignored" data.' (gh-31239) https://github.com/python/cpython/commit/cb68788dcadf43b47292bab7816a5ed9efa69730 -- ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40255] Fixing Copy on Writes from reference counting and immortal objects
Eric Snow added the comment: @Eddie, what can I do to push this forward? FYI, in addition to the python-dev thread a few weeks back, I've brought the matter up with the steering council. [1] Also, if we can get back to performance-neutral (currently at about 4% slower) then there would be a lot less controversy. Even at 2% it may be enough. [1] https://github.com/python/steering-council/issues/103 -- ___ Python tracker <https://bugs.python.org/issue40255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Change by Eric Snow : -- pull_requests: +29409 pull_request: https://github.com/python/cpython/pull/31239 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Eric Snow added the comment: New changeset 77bab59c8a1f04922bb975cc4f11e5323d1d379d by Eric Snow in branch 'main': bpo-36876: Update the c-analyzer whitelist. (gh-31225) https://github.com/python/cpython/commit/77bab59c8a1f04922bb975cc4f11e5323d1d379d -- ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36876] [subinterpreters] Global C variables are a problem
Change by Eric Snow : -- pull_requests: +29396 pull_request: https://github.com/python/cpython/pull/31225 ___ Python tracker <https://bugs.python.org/issue36876> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 81c72044a181dbbfbf689d7a977d0d99090f26a8 by Eric Snow in branch 'main': bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. (gh-30928) https://github.com/python/cpython/commit/81c72044a181dbbfbf689d7a977d0d99090f26a8 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45952] Tools/c-analyzer is out-of-date.
Eric Snow added the comment: New changeset c018d3037b5b62e6d48d5985d1a37b91762fbffb by Eric Snow in branch 'main': bpo-45952: Get the C analyzer tool working again. (gh-31220) https://github.com/python/cpython/commit/c018d3037b5b62e6d48d5985d1a37b91762fbffb -- ___ Python tracker <https://bugs.python.org/issue45952> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45952] Tools/c-analyzer is out-of-date.
Change by Eric Snow : -- pull_requests: +29390 pull_request: https://github.com/python/cpython/pull/31220 ___ Python tracker <https://bugs.python.org/issue45952> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45952] Tools/c-analyzer is out-of-date.
Eric Snow added the comment: New changeset 1e6214dbd6a980b47123229aefd60bb2c9341b53 by Eric Snow in branch 'main': bpo-45952: Get the C analyzer tool working again. (gh-31219) https://github.com/python/cpython/commit/1e6214dbd6a980b47123229aefd60bb2c9341b53 -- ___ Python tracker <https://bugs.python.org/issue45952> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45952] Tools/c-analyzer is out-of-date.
Change by Eric Snow : -- pull_requests: +29389 pull_request: https://github.com/python/cpython/pull/31219 ___ Python tracker <https://bugs.python.org/issue45952> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: (thanks Victor: https://mail.python.org/archives/list/python-...@python.org/message/7RMLIJHUWVBZFV747TFEHOE6LNBVQSMM/) 3rd party use of _Py_IDENTIFIER(): * blender + https://github.com/blender/blender/blob/master/source/blender/python/intern/bpy_traceback.c#L53 - copied from core code - "msg", "filename", "lineno", "offset", "text", "" - uses _PyObject_GetAttrId() * datatable + https://github.com/h2oai/datatable/blob/45a87337bc68576c7fb6900f524925d4fb77d6a6/src/core/python/obj.cc#L76 - in C++ wrapper getting sys.stdout, etc. and writing to sys.stdout - has to hack around C++14 support - has a fallback under limited API - "stdin", "stdout", "stderr", "write" - uses _PySys_GetObjectId(), _PyObject_GetAttrId() * multidict (in aiohttp) + https://github.com/aio-libs/multidict/blob/6dedb623cca8e8fe64f502dfa479826efc321385/multidict/_multilib/defs.h#L8 + https://github.com/aio-libs/multidict/blob/6dedb623cca8e8fe64f502dfa479826efc321385/multidict/_multilib/istr.h#L46 + https://github.com/aio-libs/multidict/blob/6dedb623cca8e8fe64f502dfa479826efc321385/multidict/_multilib/pair_list.h#L114 - calling str.lower() - "lower" - uses _PyObject_CallMethodId() * mypy (exclusively in mypyc, including in generated code!) + https://github.com/python/mypy/blob/3c935bdd1332672f5daeae7f3f9a858a45d4/mypyc/lib-rt/dict_ops.c#L76 + https://github.com/python/mypy/blob/3c935bdd1332672f5daeae7f3f9a858a45d4/mypyc/lib-rt/dict_ops.c#L131 - "setdefault", "update" - uses _PyObject_CallMethodIdObjArgs(), _PyObject_CallMethodIdOneArg() + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/pythonsupport.h#L26 + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/pythonsupport.h#L109 - "__mro_entries__", "__init_subclass__" - uses _PyObject_LookupAttrId(), _PyObject_GetAttrId() + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/misc_ops.c#L27 + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/misc_ops.c#L47 - "send", "throw", "close" - uses _PyObject_CallMethodIdOneArg(), _PyObject_GetAttrId() + https://github.com/python/mypy/blob/8c5c915a89ec0f35b3e07332c7090e62f143043e/mypyc/lib-rt/bytes_ops.c#L104 - "join" - uses _PyObject_CallMethodIdOneArg() + https://github.com/python/mypy/blob/3c935bdd1332672f5daeae7f3f9a858a45d4/mypyc/codegen/emitwrapper.py#L326 + https://github.com/python/mypy/blob/2b7e2df923f7e4a3a199915b3c8563f45bc69dfa/mypyc/lib-rt/misc_ops.c#L694 - uses _PyObject_GetAttrId() * pickle5 + https://github.com/pitrou/pickle5-backport/blob/e6117502435aba2901585cc6c692fb9582545f08/pickle5/_pickle.c#L224 + https://github.com/pitrou/pickle5-backport/blob/e6117502435aba2901585cc6c692fb9582545f08/pickle5/compat.h - "getattr" - uses _PyUnicode_FromId() * pysqlite3 + https://github.com/coleifer/pysqlite3/blob/f302859dc9ddb47a1089324dbca3873740b74af9/src/microprotocols.c#L103 + https://github.com/coleifer/pysqlite3/blob/f302859dc9ddb47a1089324dbca3873740b74af9/src/microprotocols.c#L119 - "__adapt__", "__conform__" - uses _PyObject_CallMethodId() + https://github.com/coleifer/pysqlite3/blob/093b88d1a58b141db8cf971c35ea1f6b674d0d02/src/connection.c#L51 + https://github.com/coleifer/pysqlite3/blob/093b88d1a58b141db8cf971c35ea1f6b674d0d02/src/connection.c#L772 - "finalize", "value", "upper", "cursor" - uses _PyObject_CallMethodId(), _PyObject_CallMethodIdObjArgs() + https://github.com/coleifer/pysqlite3/blob/49ce9c7a89a3c9f47ab8d32b6c4e2f7d629c1688/src/module.c#L195 - "upper" - uses _PyObject_CallMethodId() + https://github.com/coleifer/pysqlite3/blob/91b2664f525b19feedfca3f0913302c6f1e8be8a/src/cursor.c#L103 - "upper" - uses _PyObject_CallMethodId() * typed_ast + a fork of CPython's ast code * zodbpickle + a fork of CPython's pickle All of them should be trivial to drop _Py_IDENTIFIER() without any real performance impact or mess. Also, the following implies that PyPy has some sort of _Py_IDENTIFIER() support: https://github.com/benhoyt/scandir/blob/3396aa4155ffde8600a0e9ca50d5872569169b5d/_scandir.c#L51. -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Eric Snow added the comment: > This means that ThreadPoolExecutor's atexit runs before mine, > and since I never get a chance to cancel my tasks, it deadlocks. (assuming we want to support long-running tasks here) With all the above in mind, there are a few things that may help. The first that comes to mind is to have the atexit handler call ThreadPoolExecutor.shutdown() for each instance. So something like this: def _python_exit(): global _shutdown with _global_shutdown_lock: _shutdown = True for executor in list(_executors): executor.shutdown() That would require a little refactoring to make it work. However, the change is simpler if each executor has its own atexit handler: class ThreadPoolExecutor(_base.Executor): def __init__(self, ...): ... threading._register_atexit(self._atexit()) def _atexit(self): global _shutdown _shutdown = True self.shutdown() The value of either approach is that you can then subclass ThreadPoolExecutor to get what you want: class MyExecutor(ThreadPoolExecutor): def shutdown(self, *args, **kwargs): stop_my_tasks() super().shutdown(*args, **kwwargs) One thing I thought about was supporting a per-task finalizer instead, since that aligns more closely with the way ThreadPoolExecutor works. It would only apply So something like one of the following: * ThreadPoolExecutor(finalize_task=) * ThreadPoolExecutor.submit(finalize=) * ThreadPoolExecutor.register_atexit() * (classmethod) ThreadPoolExecutor.register_atexit() * concurrent.futures.register_atexit() (It would probably make sense to pass the list of currently running tasks to the callable.) -- ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Eric Snow added the comment:
FWIW, here's a brain dump about ThreadPoolExecutor and its atexit handler after
having looked at the code.
First, the relationship between the objects involved:
* work item -> Future
* work item -> task (e.g. function)
* queue -> [work item]
* worker -> executor
* worker -> queue
* worker -> currently running work item
* thread -> worker
* ThreadPoolExecutor -> [thread]
* ThreadPoolExecutor -> queue
* global state -> {thread: queue}
Observations:
* a work item's future and task are not aware of each other, and operations on
either have no effect on the other
Next, let's look at the relevant ways the objects can be used:
* publicly
* ThreadPoolExecutor.submit(task) -> Future
* ThreadPoolExecutor.shutdown()
* Future.result() and Future.exception()
* Future.cancel()
* Future.add_done_callback()
* internally
* queue.pop() -> work item
* .run()
* thread.join()
* Future.set_running_or_notify_cancel()
* Future.set_result() and Future.set_exception()
Observations:
* nothing interacts with a worker directly; it is waited on via its thread and
it receives work (or None) via the queue it was given
* once a worker pops the next work item off the queue, nothing else has access
to that work item (the original ThreadPoolExecutor().submit() caller has the
Future, but that's it)
* there is no cancelation mechanism for tasks
* there is no simple way to interact with the queued tasks
* realistically, there is no way to interact with the currently running task
* there is no reliable way to "kill" a thread
Regarding how tasks run:
* ThreadPoolExecutor.submit(task) -> Future
* ThreadPoolExecutor.submit(task) -> work item (Future, task) -> queue
* ThreadPoolExecutor.submit(task) -> thread (worker)
* thread -> worker -> ( queue -> work item -> task )
Observations::
* the worker loop exits if the next item in the queue is None (and the executor
is shutting down)
Now lets look more closely at the atexit handler.
* as you noted, since 3.9 it is registered with threading._register_atexit()
instead of atexit.register()
* the threading atexit handlers run before the regular atexit handlers
* the ThreadPoolExecutor handler does not actually interact with
ThreadPoolExecutor instances directly
* it only deals with a module-global list of (thread, [work item]) pairs, to
which ThreadPoolExecutor instances add items as they go
The handler does the following:
1. disables ThreadPoolExecutor.submit() (for all instances)
2. (indirectly) tells each worker to stop ASAP
3. lets every pending task run (and lets every running task keep running)
4. waits until all tasks have finished
It does not:
* call any ThreadPoolExecutor.shutdown()
* otherwise deal with the ThreadPoolExecutor instances directly
* call Future.cancel() for any of the tasks
* use any timeout in step 4, so it may block forever
* notify tasks that they should finish
* deal well with any long-running (or infinite) task
ThreadPoolExecutor.shutdown() basically does the same thing. However, it only
does the steps above for its own tasks. It also optionally calls
Future.cancel() for each queued task (right before step 2). However, all that
does is keep any queued-but-not-running tasks from starting. Also, you can
optionally skips step 4.
--
___
Python tracker
<https://bugs.python.org/issue41962>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Eric Snow added the comment: > I'm running some long-running (possibly infinite) tasks in the thread pool, > and I cancel them in an `atexit` callback Alternately, perhaps ThreadPoolExecutor isn't the right fit here, as implied by the route you ended up going. It seems like it's not well-suited for long-running (or infinite) tasks. In that case, perhaps the concurrent.futures docs could be more clear about when ThreadPoolExecutor is not a good fit and what the alternatives are. -- ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41962] Make threading._register_atexit public?
Eric Snow added the comment: > I'm running some long-running (possibly infinite) tasks in the thread pool, > and I cancel them in an `atexit` callback To be clear, by "cancel" you are not talking about Future.cancel(). Rather, your handler causes all running tasks to finish (by sending a special message on the socket corresponding to each running task). Is that right? Some other things I inferred from your atexit handler: * it does not make sure the task associated with the socket finishes (no way of knowing?) * so if a task hangs while trying to stop then the running thread in the ThreadPoolExecutor would block shutdown forever * similarly, if a task is stuck handling a request then it will never receive the special message on the socket, either blocking the send() in your handler or causing ThreadPoolExecutor shutdown/atexit to wait forever * it vaguely implies a 1-to-1 relationship between sockets and *running* tasks * likewise that pending (queued) tasks do not have an associated socket (until started) * so once your handler finishes, any tasks pending in the ThreadPoolExecutor queue will eventually get started but never get stopped by your handler; thus you're back to the deadlock situation Does all that sound right? Most of that is relevant to some possible solutions I have in mind. -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue41962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: FYI, I've posted to python-dev for feedback before proceeding: https://mail.python.org/archives/list/python-...@python.org/thread/DNMZAMB4M6RVR76RDZMUK2WRLI6KAAYS/ -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: If necessary, we can keep _Py_IDENTIFIER() (and the functions). Regardless, we can stop using it internally. -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +29221 pull_request: https://github.com/python/cpython/pull/31038 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: > Any chance we could revert the recent renaming of tstate.exc_state and > tstate.root_cframe Yeah, I'll sort this out. Sorry for that. -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46524] test_peg_generator takes 8 minutes on Windows
Eric Snow added the comment: On Tue, Jan 25, 2022 at 4:14 PM STINNER Victor wrote: > Currently, most CI run "make buildbottest" which uses -r option of > libregrtest: randomize tests order. How hard would it be to first randomize the list and then move the slow tests up to a random position in the first half (for example) of the list? -- nosy: +eric.snow ___ Python tracker <https://bugs.python.org/issue46524> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: New changeset 247480a21cb165efdacc346a2d589dfc27e18283 by Eric Snow in branch 'main': bpo-46541: Generate the global objects initializer. (gh-30941) https://github.com/python/cpython/commit/247480a21cb165efdacc346a2d589dfc27e18283 -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46166] Get "self" args or non-null co_varnames from frame object with C-API
Eric Snow added the comment: In addition to what Mark said, note that co_varnames get's populated lazily by the Python-level getter for code.co_varnames. So could you call the Python function before entering the hot path? Regardless, a dedicated C-API for this like Mark suggested would be the better solution. -- ___ Python tracker <https://bugs.python.org/issue46166> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Change by Eric Snow : -- keywords: +patch pull_requests: +29120 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/30941 ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
Eric Snow added the comment: ## Background ## `_Py_Identifier` (and `_Py_IDENTIFIER()`, etc.) was added in 2011 [1][2] for several reasons: * provide a consistent approach for a common optimization: caching C-string based string objects * facilitate freeing those objects during runtime finalization The solution involved using a static variable defined, using `_Py_IDENTIFIER()` near the code that needed the string. The variable (a `_Py_Identifier`) would hold the desired C string (statically initialized) and the corresponding (lazily created) `PyUnicodeObject`. The code where the `_Py_Identifier` was defined would then pass it to specialized versions of various C-API that would normally consume a C string or `PyUnicodeObject`. Then that code would use either the C-string or the object (creating and caching it first if not done already). This approach decentralized the caching but also provided a single tracking mechanism that made it easier to clean up the objects. Over the last decade a number of changes were made, including recent changes to make the identifiers per-interpreter and to use a centralized cache. [1] https://github.com/python/cpython/commit/afe55bba33a20f87a58f940186359237064b428f [2] https://mail.python.org/archives/list/python-...@python.org/message/FRUTTE47JO2XN3LXV2J4VB5A5VILILLA/ -- ___ Python tracker <https://bugs.python.org/issue46541> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46541] Replace _Py_IDENTIFIER() with statically initialized objects.
New submission from Eric Snow :
`_Py_Identifier` has been useful but at this point there is a faster and
simpler approach we could take as a replacement: statically initialize the
objects as fields on `_PyRuntimeState` and reference them directly through a
macro.
This would involve the following:
* add a `PyUnicodeObject field (not a pointer) to `_PyRuntimeState` for each
string that currently uses `_Py_IDENTIFIER()`
* initialize each object as part of the static initializer for `_PyRuntimeState`
* make each "immortal" (e.g. start with a really high refcount)
* add a macro to look up a given string
* update each location that currently uses `_Py_IDENTIFIER()` to use the new
macro instead
As part of this, we would also do the following:
* get rid of all C-API functions with `_Py_Identifer` parameters
* get rid of the old runtime state related to identifiers
* get rid of `_Py_Identifier`, `_Py_IDENTIFIER()`, etc.
(Note that there are several hundred uses of `_Py_IDENTIFIER()`, including a
number of duplicates.)
Pros:
* reduces indirection (and extra calls) for C-API using the strings (making the
code easier to understand and speeding it up)
* the objects are referenced from a fixed address in the static data section
(speeding things up and allowing the C compiler to optimize better)
* there is no lazy allocation (or lookup, etc.) so there are fewer possible
failures when the objects get used (thus less error return checking)
* simplifies the runtime state
* saves memory (at little, at least)
* the approach for per-interpreter is simpler (if needed)
* reduces the number of static variables in any given C module
* reduces the number of functions in the ("private") C-API
* "deep frozen" modules can use these strings
* other commonly-used strings could be pre-allocated by adding
`_PyRuntimeState` fields for them
Cons:
* churn
* adding a string to the list requires modifying a separate file from the one
where you actually want to use the string
* strings can get "orphaned" (we could prevent this with a check in `make
check`)
* some PyPI packages may rely on `_Py_IDENTIFIER()` (even though it is
"private" C-API)
* some strings may never get used for any given ./python invocation
Note that with a basic partial implementation (GH-30928) I'm seeing a 1%
improvement in performance (see
https://github.com/faster-cpython/ideas/issues/230).
--
assignee: eric.snow
components: Interpreter Core
messages: 411799
nosy: eric.snow, serhiy.storchaka, vstinner
priority: normal
pull_requests: 29107
severity: normal
stage: needs patch
status: open
title: Replace _Py_IDENTIFIER() with statically initialized objects.
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue46541>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46449] Deep-freezed modules create inconsistency in sys.gettotalrefcount() (_Py_Reftotal)
Eric Snow added the comment: > the deep-frozen objects also reference the small ints directly, as well as > the singleton for b"". > Is this even safe across Py_Finalize()/Py_Initialize()? If not, we'll need to > roll that back as well. The small ints and the empty bytes object each have "immortal" refcounts too (9, just like you did in deepfreeze). So they would cause a similar behavior to what Victor reported. Otherwise I wouldn't expect any problems across Py_Finalize()/Py_Initialize(). -- ___ Python tracker <https://bugs.python.org/issue46449> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: New changeset 322f962f3ee31d0dbde99e36379de8488ccc6804 by Eric Snow in branch 'main': bpo-45953: Statically initialize all the non-object PyInterpreterState fields we can. (gh-30589) https://github.com/python/cpython/commit/322f962f3ee31d0dbde99e36379de8488ccc6804 -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Eric Snow added the comment: New changeset 324908ba936d5d262026deebb81f050803848c41 by Eric Snow in branch 'main': bpo-45953: Statically initialize all the PyThreadState fields we can. (gh-30590) https://github.com/python/cpython/commit/324908ba936d5d262026deebb81f050803848c41 -- ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +28788 pull_request: https://github.com/python/cpython/pull/30590 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +28787 pull_request: https://github.com/python/cpython/pull/30589 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45953] Statically allocate interpreter states as much as possible.
Change by Eric Snow : -- pull_requests: +28786 pull_request: https://github.com/python/cpython/pull/30588 ___ Python tracker <https://bugs.python.org/issue45953> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46370] Move runtime static init to its own header file.
Change by Eric Snow : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue46370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46370] Move runtime static init to its own header file.
Eric Snow added the comment: New changeset bc02eac9d2cb36faffc5027b7ce09e6dd0922a7f by Eric Snow in branch 'main': bpo-46370: Move the static initializer for _PyRuntime to its own header file. (gh-30587) https://github.com/python/cpython/commit/bc02eac9d2cb36faffc5027b7ce09e6dd0922a7f -- ___ Python tracker <https://bugs.python.org/issue46370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46370] Move runtime static init to its own header file.
Change by Eric Snow : -- keywords: +patch pull_requests: +28785 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/30587 ___ Python tracker <https://bugs.python.org/issue46370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46370] Move runtime static init to its own header file.
New submission from Eric Snow : The static initializer for `_PyRuntime` is currently defined in Include/internal/pycore_runtime.h. However, it is only needed by Python/pylifecycle.c (and Python/pystate.c for an optimization) and should only be used there. (Also, the initializer is quite large.) So I'm planning on moving it to it's own internal header file. -- assignee: eric.snow components: Interpreter Core messages: 410529 nosy: eric.snow priority: normal severity: normal stage: needs patch status: open title: Move runtime static init to its own header file. versions: Python 3.11 ___ Python tracker <https://bugs.python.org/issue46370> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46360] Inconsistent import behavior for (unusual) submodules
Eric Snow added the comment: > I'm going to assume the "even though sys.modules has `None`" case, > which I think is an oversight and should probably get fixed Yep, I agree. That's the case I was looking at in the first place. I noticed the other two as I was hacking together code to verify the None behavior. :) Bothering to change those would be more trouble than its worth. -- ___ Python tracker <https://bugs.python.org/issue46360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46070] [subinterpreters] crash when importing _sre in subinterpreters in parallel (Python 3.9 regression)
Eric Snow added the comment: > (*) I made the GC state per-interpreter: commit > 7247407c35330f3f6292f1d40606b7ba6afd5700 (Nov 20, 2019) FYI, this was done by me in an earlier comment which we ended up reverting. Later you basically un.reverted that. > The bug is that a C function object (_sre.compile) is created in an > interpreter, tracked by the GC list of this interpreter, and then it is > destroye and untracked in another interpreter. FWIW, at one point I had a branch that supported sharing read-only Py_Buffer data. When the receiving interpreter was done with it I'd call Py_AddPendingCall() to schedule the cleanup in the "owner" interpreter. However, this only worked because I kept track of the owner. Adding that pointer to every object wouldn't be feasible but I suppose there are other things we could do that wouldn't be super inefficient, like don't worry about it for the main interpreter, use a hash table (Victor's idea), borrow some of the bits of the PyObject head to store a flag or even an index into an array (if there are only a few interpreters), or even make the allocator per-interpreter and then extrapolate the interpreter based on the object's address. Regardless, it is still much simpler to make all objects per-interpreter. -- ___ Python tracker <https://bugs.python.org/issue46070> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
