[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by STINNER Victor : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by Pablo Galindo Salgado : -- priority: release blocker -> ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: _PyUnicode_EqualToASCIIId() and type update_slot() functions are fixed in 3.10 and main branches. The regression is now fixed. But the revert reintroduces the issue on subinterpreters, so I created bpo-46283: "[subinterpreters] Unicode interned strings must

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: New changeset 72c260cf0c71eb01eb13100b751e9d5007d00b70 by Victor Stinner in branch '3.10': [3.10] bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (GH-20085)" (GH-30422) (GH-30425)

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by STINNER Victor : -- pull_requests: +28639 pull_request: https://github.com/python/cpython/pull/30433 ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by STINNER Victor : -- pull_requests: +28630 pull_request: https://github.com/python/cpython/pull/30425 ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: New changeset 35d6540c904ef07b8602ff014e520603f84b5886 by Victor Stinner in branch 'main': bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (GH-20085)" (GH-30422)

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Eric Snow added the comment: > IMO writing a complete rationale for running multiple interpreters in > parallel which require a whole PEP. FYI, I'm planning on having such a PEP published in the next few days. -- ___ Python tracker

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Eric Snow added the comment: +1 on just reverting in both branches. I can deal with my stuff separately. -- ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: IMO writing a complete rationale for running multiple interpreters in parallel which require a whole PEP. I didn't write such PEP yet since there are still non-trivial technical issues, especially the problem of static types: bpo-40601. I don't have time

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by STINNER Victor : -- pull_requests: +28627 pull_request: https://github.com/python/cpython/pull/30422 ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Petr Viktorin added the comment: > Personally, I'd rather we not revert the original change. Even in 3.10? Your PR looks pretty heavy for a bugfix release, and won't apply cleanly to 3.10. > Moving the data to _PyRuntimeState would save me some effort with related > work I'm doing right

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Eric Snow added the comment: are there any objections to my PR? -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by Sebastian Rittau : -- nosy: +srittau ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Eric Snow added the comment: If that seems okay, I'll work on a backport PR for 3.10. -- ___ Python tracker ___ ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Eric Snow added the comment: I've created a PR for moving the interned strings and identifiers to _PyRuntimeState until we are ready to move them back to the interpreter. -- ___ Python tracker

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by Eric Snow : -- pull_requests: +28350 pull_request: https://github.com/python/cpython/pull/30131 ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Eric Snow added the comment: FWIW, it makes sense to me for the interned strings to be per-interpreter eventually. Otherwise strings interned by an interpreter would persist after that interpreter is finalized, potentially leaking memory until the runtime is finalized. However, if we end up

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Eric Snow added the comment: It sounds like this bug is another case where we have made some objects per-interpreter but others are still global and this is causing problems. _PyUnicode_EqualToASCIIId() wouldn't have any problems if interpreters weren't sharing any objects (or were only

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: Mark: "As Victor points out, there is no bug in 3.9 because interned strings are common across all interpreter. We should revert that behavior." The rationale for having per-interpreter interned strings and per-interpreter _Py_IDENTIFIER() string objects

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Mark Shannon added the comment: The problem here is that different sub-interpreters have different strings for the same Python string. Unless sub-interpreters are fully independent, and they cannot be due to limitations imposed by the stable API, then all sub-interpreters must share the

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: See also bpo-46070: I don't know if it's related. -- ___ Python tracker ___ ___ Python-bugs-list

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: These bug prevent the Fedora infra team from upgrading the Koji builders to Fedora 35. Koji runs on mod_wsgi which is affected by the bug: https://bugzilla.redhat.com/show_bug.cgi?id=2030621#c1 -- ___ Python

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: I created PR 30123 to fix _PyUnicode_EqualToASCIIId() and type update_slot() functions. I added comments explaining why we can no longer optimize the comparison of two interned string objects. -- ___ Python

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by STINNER Victor : -- pull_requests: +28342 stage: -> patch review pull_request: https://github.com/python/cpython/pull/30123 ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by Craig Holmquist : -- nosy: +craigh ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by Jonas Witschel : -- nosy: +diabonas ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by Miro Hrončok : -- nosy: +hroncok ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by STINNER Victor : -- keywords: +3.10regression nosy: +pablogsal priority: normal -> release blocker ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: Attachd cmp_interned_strings.patch fix _PyUnicode_EqualToASCIIId() and update_slot() for subinterpreters. I will create a PR once we agree if it's required to support subinterpreters there, and if there is no better (faster) option. For update_slot(), one

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: I marked bpo-46034 as a duplicate: setting a class "__init__" attribute doesn't update properly its tp_init slot. update_slot() of Objects/typeobject.c only uses a fast pointer comparison since both strings are interned, but the test fails if the two

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Dong-hee Na added the comment: > If two strings are interned and part of the same interpreter, the "ptr1 == > ptr2" comparison continues to work. Yeah, AFAIK Comparing two interned strings from different interpreters are not the available use-case. --

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: > We can not compare two Unicode by pointer even if both are interned > anymore... It was a nice optimization. If two strings are interned and part of the same interpreter, the "ptr1 == ptr2" comparison continues to work. What is not compatible with

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Inada Naoki added the comment: That's too bad. We can not compare two Unicode by pointer even if both are interned anymore... It was a nice optimization. -- ___ Python tracker

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Change by Nathan Jensen : -- nosy: +ndjensen ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: > Should `_PyUnicode_EqualToASCIIId()` support comparing two unicode from > different interpreter? Right now, there still many cases where objects are still shared between two interpreters: * None, True, False singletons * strings from code objects

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

Inada Naoki added the comment: Should `_PyUnicode_EqualToASCIIId()` support comparing two unicode from different interpreter?? -- nosy: +methane ___ Python tracker ___

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: There are around 27 _PyUnicode_EqualToASCIIId() calls in the Python code base. I don't think that avoiding _PyUnicode_EqualToASCIIId() is a good solution :-) Fixing _PyUnicode_EqualToASCIIId() to make it compatible with subinterpreters sound more

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: > Depending on how the stdlib abc.py file was loaded (in the main interpreter > and in the subinterpreter), __code__.co_freevars[0] may or may not be an > interned string. When the bug occurs, I see that the Python stdlib abc.py file is loaded twice: the

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: Serhiy: Do you recall the idea of the PyUnicode_CHECK_INTERNED() optimization? The PyUnicode_CHECK_INTERNED() test is as old as the _PyUnicode_EqualToASCIIId() function. commit f5894dd646f5e39918377b37b8c8694cebdca103 Author: Serhiy Storchaka Date: Wed

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

STINNER Victor added the comment: In Python 3.9, the code works because the _Py_IDENTIFIER() API shares Python Unicode objects between all interpreters. _PyUnicode_FromId() was modified to be per-interpreter in bpo-39465 by: New changeset ba3d67c2fb04a7842741b1b6da5d67f22c579f33 by Victor

[issue46006] [subinterpreter] _PyUnicode_EqualToASCIIId() issue with subinterpreters

New submission from STINNER Victor : _PyUnicode_EqualToASCIIId() seems to be incompatible with subinterpreter: it makes the assumption that if direct pointer comparison fails and the string is interned, the two strings are not equal. -- super_init_without_args() of Objects/typeobject.c