On 09. 03. 22 4:38, Eric Snow wrote:
I'd really appreciate feedback on this new PEP about making the GIL
per-interpreter.
Yay! Thank you!
This PEP definitely makes per-interpreter GIL sound possible :)
The PEP targets 3.11, but we'll see if that is too close. I don't
mind waiting one more
release, though I'd prefer 3.11 (obviously). Regardless, I have no
intention of rushing
this through at the expense of cutting corners. Hence, we'll see how it goes.
> The PEP text is included inline below. Thanks!
-eric
===================================================
PEP: 684
Title: A Per-Interpreter GIL
Author: Eric Snow <ericsnowcurren...@gmail.com>
Discussions-To: python-dev@python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
This iteration of the PEP should also have `Requires: 683` (Immortal
Objects).
[...]
Motivation
==========
The fundamental problem we're solving here is a lack of true multi-core
parallelism (for Python code) in the CPython runtime. The GIL is the
cause. While it usually isn't a problem in practice, at the very least
it makes Python's multi-core story murky, which makes the GIL
a consistent distraction.
Isolated interpreters are also an effective mechanism to support
certain concurrency models. :pep:`554` discusses this in more detail.
Indirect Benefits
-----------------
Most of the effort needed for a per-interpreter GIL has benefits that
make those tasks worth doing anyway:
* makes multiple-interpreter behavior more reliable
* has led to fixes for long-standing runtime bugs that otherwise
hadn't been prioritized > * has been exposing (and inspiring fixes for) previously unknown
runtime bugs
* has driven cleaner runtime initialization (:pep:`432`, :pep:`587`)
* has driven cleaner and more complete runtime finalization
* led to structural layering of the C-API (e.g. ``Include/internal``)
* also see `Benefits to Consolidation`_ below
Do you want to dig up some bpo examples, to make these more convincing
to the casual reader?
Furthermore, much of that work benefits other CPython-related projects:
* performance improvements ("faster-cpython")
* pre-fork application deployment (e.g. Instagram)
Maybe say “e.g. with Instagram's Cinder” – both the household name and
the project you can link to?
* extension module isolation (see :pep:`630`, etc.)
* embedding CPython
A lot of these points are duplicated in "Benefits to Consolidation" list
below, maybe there'd be, ehm, benefits to consolidating them?
[...]
PEP 554
-------
Please spell out "PEP 554 (Multiple Interpreters in the Stdlib)", for
people who don't remember the magic numbers but want to skim the table
of contents.
:pep:`554` is strictly about providing a minimal stdlib module
to give users access to multiple interpreters from Python code.
In fact, it specifically avoids proposing any changes related to
the GIL. Consider, however, that users of that module would benefit
from a per-interpreter GIL, which makes PEP 554 more appealing.
Rationale
=========
During initial investigations in 2014, a variety of possible solutions
for multi-core Python were explored, but each had its drawbacks
without simple solutions:
* the existing practice of releasing the GIL in extension modules
* doesn't help with Python code
* other Python implementations (e.g. Jython, IronPython)
* CPython dominates the community
* remove the GIL (e.g. gilectomy, "no-gil")
* too much technical risk (at the time)
* Trent Nelson's "PyParallel" project
* incomplete; Windows-only at the time
* ``multiprocessing``
* too much work to make it effective enough;
high penalties in some situations (at large scale, Windows)
* other parallelism tools (e.g. dask, ray, MPI)
* not a fit for the stdlib
* give up on multi-core (e.g. async, do nothing)
* this can only end in tears
This list doesn't render correctly in ReST, you need blank lines everywhere.
There are more cases like this below.
[...]> Per-Interpreter State
---------------------
The following runtime state will be moved to ``PyInterpreterState``:
* all global objects that are not safely shareable (fully immutable)
* the GIL
* mutable, currently protected by the GIL
Spelling out “mutable state” in these lists would make this clearer,
since “state” isn't elided from all the points.
* mutable, currently protected by some other per-interpreter lock
* mutable, may be used independently in different interpreters
This includes extension modules (with multi-phase init), right?
* all other mutable (or effectively mutable) state
not otherwise excluded below
Furthermore, a number of parts of the global state have already been
moved to the interpreter, such as GC, warnings, and atexit hooks.
The following state will not be moved:
* global objects that are safely shareable, if any
* immutable, often ``const``
* treated as immutable
Do you have an example for this?
* related to CPython's ``main()`` execution
* related to the REPL
Would “only used by” work instead of “related to”?
* set during runtime init, then treated as immutable
`main()`, REPL and runtime init look like special cases of functionality
that only runs in one interpreter. If it's so, maybe generalize this?
[...]
C-API
-----
The following private API will be made public:
* ``_PyInterpreterConfig``
* ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``)
Since the API is not documented (and _PyInterpreterConfig is not even in
main yet!), it would be good to sketch out the docs (intended behavior)
here.
The following fields will be added to ``PyInterpreterConfig``:
* ``own_gil`` - (bool) create a new interpreter lock
(instead of sharing with the main interpreter)
As a user of the API, what should I consider when setting this flag?
Would the GIL be shared with the *parent* interpreter or the main one?
What are the restrictions/implications of this flag?
* ``strict_extensions`` - fail import in this interpreter for
incompatible extensions (see `Restricting Extension Modules`_)
I'm not sure about including a workaround flag in the structure.
Since the Python API will get a context manager for this, maybe the C
API should get a function to set/reset it instead of this flag?
Restricting Extension Modules
-----------------------------
Extension modules have many of the same problems as the runtime when
state is stored in global variables. :pep:`630` covers all the details
of what extensions must do to support isolation, and thus safely run in
multiple interpreters at once. This includes dealing with their globals.
Extension modules that do not implement isolation will only run in
the main interpreter. In all other interpreters, the import will
raise ``ImportError``. This will be done through
``importlib._bootstrap_external.ExtensionFileLoader``.
“Main interpreter” should be defined. (Or maybe the term should be
avoided instead -- always having to spell out “interpreter started by
Py_Initialize rather than Py_NewInterpreter” might push us toward
finding ways to avoid the special case...)
We will work with popular extensions to help them support use in
multiple interpreters. This may involve adding to CPython's public C-API,
which we will address on a case-by-case basis.
Extension Module Compatibility
''''''''''''''''''''''''''''''
As noted in `Extension Modules`_, many extensions work fine in multiple
interpreters without needing any changes. The import system will still
fail if such a module doesn't explicitly indicate support. At first,
not many extension modules will, so this is a potential source
of frustration.
We will address this by adding a context manager to temporarily disable
the check on multiple interpreter support:
``importlib.util.allow_all_extensions()``. >
I'd prefer a more dangerous-sounding name, to guide code readers (and
autocomplete users) toward checking the warning in the docs.
[...]
Extension Modules
'''''''''''''''''
Currently the most common usage of Python, by far, is with the main
interpreter running by itself. This proposal has zero impact on
extension modules in that scenario. Likewise, for better or worse,
there is no change in behavior under multiple interpreters created
using the existing ``Py_NewInterpreter()``.
Keep in mind that some extensions already break when used in multiple
interpreters, due to keeping module state in global variables. They
may crash or, worse, experience inconsistent behavior. That was part
of the motivation for :pep:`630` and friends, so this is not a new
situation nor a consequence of this proposal.
In contrast, when the `proposed API <proposed capi_>`_ is used to
create multiple interpreters, the default behavior will change for
some extensions. In that case, importing an extension will fail
(outside the main interpreter) if it doesn't indicate support for
multiple interpreters. For extensions that already break in
multiple interpreters, this will be an improvement.
Now we get to the break in compatibility mentioned above. Some
extensions are safe under multiple interpreters, even though they
haven't indicated that. Unfortunately, there is no reliable way for
the import system to infer that such an extension is safe, so
importing them will still fail. This case is addressed in
`Extension Module Compatibility`_ below.
Extensions that use multi-phase init should already be compatible with
multiple interpreters. Multi-phase init itself is the flag that
indicates this.
But they might not be compatible with *per-interpreter GIL*. I don't
like how that's conflated with multiple interpreters here.
For example, extension modules can currently support multiple
interpreters, but rely on the GIL to protect calls to a non-threadsafe
library, access shared memory, etc. As an example, the PEP 630 “opt-out”
is not thread-safe.
It seems to me that there should be a separate flag (slot) to indicate
support for per-interpreter GIL, and the `strict_extensions` bit should
work with that.
[...]
How to Teach This
=================
This is an advanced feature for users of the C-API. There is no
expectation that this will be taught.
Oh, I'm afraid this will need some docs related to making sure an
extension is compatible with per-interpreter GIL.
I'd rather not repeat my mistake of hand-wavingly noting "All modules
created using multi-phase initialization are expected to support
sub-interpreters" in the docs, and only writing PEP 630 much later.
[...]
References
==========
Related:
* :pep:`384`
* :pep:`432`
* :pep:`489`
* :pep:`554`
* :pep:`573`
* :pep:`587`
* :pep:`630`
* :pep:`683`
* :pep:`3121`
Please write out the titles here.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/UEUYVZ5IHSE3FYUDJ3INLSDWS7IZITOB/
Code of Conduct: http://python.org/psf/codeofconduct/
