[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[Jon Parise]

Change by Jon Parise :


--
nosy: +jon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[Nathan Jensen]

Change by Nathan Jensen :


--
nosy: +ndjensen

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[Mark Shannon]

Change by Mark Shannon :


--
nosy: +Mark.Shannon
nosy_count: 9.0 -> 10.0
pull_requests: +27492
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/29228

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[Petr Viktorin]

Petr Viktorin  added the comment:

PyStructSequence_NewType exists, and is the same as the proposed 
PyStructSequence_FromModuleAndDesc except it doesn't take the module (which 
isn't necessary: PyStructSequence_Desc has no way to define functionality that 
would need the module state).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[hai shi]

hai shi  added the comment:

bpo-45113: [subinterpreters][C API] Add a new function to create 
PyStructSequence from Heap.

--
versions: +Python 3.11 -Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

Hai Shi:
> IMO, I suggest to create a new function, PyStructSequence_FromModuleAndDesc()

Please create a new issue. If possible, I would prefer to have a sub-issue for 
that, to keep this issue as a tracking issue for all issues related to 
subinterpreters.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[hai shi]

hai shi  added the comment:

> PyStructSequence_InitType2() is not compatible with subinterpreters: it uses 
> static types. Moreover, it allocates tp_members memory which is not released 
> when the type is destroyed. But I'm not sure that the type is ever destroyed, 
> since this API is designed for static types.

IMO, I suggest to create a new function, 
PyStructSequence_FromModuleAndDesc(module, desc, flags) to create a heaptype 
and don't aloocates memory block for tp_members,something like 
'PyType_FromModuleAndSpec()`.

I don't know there have any block issue to do this converting operation. But I 
can take a look.

@petr ping, Petr, do you have any better idea about this question :)

--
nosy: +petr.viktorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

PyStructSequence_InitType2() is not compatible with subinterpreters: it uses 
static types. Moreover, it allocates tp_members memory which is not released 
when the type is destroyed. But I'm not sure that the type is ever destroyed, 
since this API is designed for static types.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

> * Add a lock to pymalloc, or disable pymalloc when subinterpreters are used: 
> https://github.com/ericsnowcurrently/multi-core-python/issues/30

See bpo-43313: "feature: support pymalloc for subinterpreters. each 
subinterpreter has pymalloc_state".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[Nicholas Sim]

Change by Nicholas Sim :


--
nosy: +nw0

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[Alex Garel]

Change by Alex Garel :


--
nosy: +alex-garel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

FYI I wrote an article about this issue: "Isolate Python Subinterpreters"
https://vstinner.github.io/isolate-subinterpreters.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

> * resolve_slotdups() uses a static variable

Attached resolve_slotdups.patch works around the issue by removing the cache.

--
keywords: +patch
Added file: https://bugs.python.org/file49700/resolve_slotdups.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

I played with ./configure --with-experimental-isolated-subinterpreters. I tried 
to run "pip list" in parallel in multiple interpreters.

I hit multiple issues:

* non-atomic reference count of Python objects shared by multiple interpreters, 
objects shared via static types for example.

* resolve_slotdups() uses a static variable

* pip requires _xxsubinterpreters.create(isolated=False): the vendored distro 
package runs the lsb_release command with subprocess.

* Race conditions in PyType_Ready() on static types:

  * Objects/typeobject.c:5494: PyType_Ready: Assertion "(type->tp_flags & (1UL 
<< 13)) == 0" failed
  * Race condition in add_subclass()

* parser_init() doesn't support subinterpreters

* unicode_dealloc() fails to delete an interned string in the Unicode interned 
dictionary => https://bugs.python.org/issue40521#msg383829


To run "pip list", I used:

CODE = """
import runpy
import sys
import traceback
sys.argv = ["pip", "list"]
try:
runpy.run_module("pip", run_name="__main__", alter_sys=True)
except SystemExit:
pass
except Exception as exc:
traceback.print_exc()
print("BUG", exc)
raise
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

> Type method cache is shared.

I created bpo-42745: "[subinterpreters] Make the type attribute lookup cache 
per-interpreter".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

See also bpo-15751: "Make the PyGILState API compatible with subinterpreters".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

FYI I'm also using https://pythondev.readthedocs.io/subinterpreters.html to 
track the progress on isolating subinterpreters.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[Erlend Egeberg Aasland]

Change by Erlend Egeberg Aasland :


--
nosy: +erlendaasland

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

> * _PyLong_Zero and _PyLong_One singletons are shared

Removed by bpo-42161 (commit c310185c081110741fae914c06c7aaf673ad3d0d).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

> Update of the EXPERIMENTAL_ISOLATED_SUBINTERPRETERS status.

Also:

* _PyLong_Zero and _PyLong_One singletons are shared
* Py_None, Py_True and Py_False singletons are shared: bpo-39511 and PR 18301
* Static types like PyUnicode_Type and PyLong_Type are shared: see bpo-40077 
and bpo-40601
* The dictionary of Unicode interned strings is shared: PR 20085
* context.c: _token_missing singleton is shared
* "struct _PyArg_Parser" generated by Argument Clinic is shared: see 
_PyArg_Fini()

Misc notes:

* init_interp_main(): if sys.warnoptions is not empty, "import warnings" is 
called to process these options, but not in subinterpreters: only in the main 
intepreter.
* _PyImport_FixupExtensionObject() contains code specific to the main 
interpreter. Maybe this function will not longer be needed once builtin 
extension modules will be converted to PEP 489 "multiphase initialization" API. 
I'm not sure.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

Update of the EXPERIMENTAL_ISOLATED_SUBINTERPRETERS status.

I made many free lists and singletons per interpreter in bpo-40521.

TODO:

* _PyUnicode_FromId() and interned strings are still shared: typeobject.c 
requires a workaround for that.
* GC is disabled in subinterpreters since some objects are still shared
* Type method cache is shared.
* pymalloc is shared.
* The GIL is shared.

I'm investigating performance of my _PyUnicode_FromId() PR: 
https://github.com/python/cpython/pull/20058

This PR now uses "atomic functions" proposed in a second PR: 
https://github.com/python/cpython/pull/20766

The "atomic functions" avoids the need to have to declare a variable or a 
structure member as atomic, which would cause different issues if they are 
declared in Python public headers (which is the case for _Py_Identifier used by 
_PyUnicode_FromId()).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

Currently, the import lock is shared by all interpreters. It would also help 
for performance to make it per-interpreter to parallelize imports.

--
versions: +Python 3.10 -Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

I created a new "Subinterpreters" component in the bug tracker. It may help to 
better track all issues related to subinterpreters.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

STINNER Victor  added the comment:

I marked bpo-36877 "[subinterpreters][meta] Move fields from _PyRuntimeState to 
PyInterpreterState" as a duplicate of this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40512] [subinterpreters] Meta issue: per-interpreter GIL

[STINNER Victor]

Change by STINNER Victor :


--
components: +Subinterpreters -Interpreter Core
title: Meta issue: per-interpreter GIL -> [subinterpreters] Meta issue: 
per-interpreter GIL

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com