[Python-Dev] Re: PEP 554 comments

[Greg Ewing]

On 21/04/20 10:35 am, Eric Snow wrote:

For the event loop case, what is the downside
to adapting to the API in the existing proposal?


If you mean the suggestion of having async send and receive
methods, that's okay.

My point was that before responding to requests to add
individual features such as buffered channels, it might be
better to provide a general mechanism that would allow
people to implement their own favourite features themselves.

It seems like you're going to need some kind of callback
mechanism under the covers to implement async send and
receive anyway, so you may as well expose it as part of
the official API.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K5Z3WUGNAVKH3GHVP7SNUL6CVXEJ4QNI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Greg Ewing]

On 21/04/20 10:47 am, Eric Snow wrote:

On Mon, Apr 20, 2020 at 4:23 PM Eric Snow  wrote:

As I've gone to update the PEP for this I'm feeling less comfortable
with changing it.


I don't get this whole business of channels being associated
with interpreters, or why there needs to be a distinction
between release() and close().

To my mind, a channel reference should be like a file
descriptor for a pipe. When you've finished with it, you
close() it. When the last reference to a channel is
closed or garbage collected, the channel disappears.
Why make it any more complicated than that?

You seem to be worried about channels getting leaked if
someone forgets to close them. But it's just the same
for files and pipes, and nobody seems to worry about
that.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D4DIHWUGC2LRYUIZ2PICOF6PZY5GKZJP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Greg Ewing]

On 21/04/20 11:24 am, Edwin Zimmerman wrote:

Socket connections could be passed off from the main interpreter to
sub-interpreters for concurrent processing that simply isn't possible
with the global GIL


I would need convincing that the processing involved things
that are truly held up by the GIL, rather than things like
blocking I/O calls that can release the GIL.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PLE7T4UXJI2XOI4C2GVT4PNFMTG7QFFY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Antoine Pitrou]

On Mon, 20 Apr 2020 15:30:37 -0700
Nathaniel Smith  wrote:
> 
> tl;dr: accepting PEP 554 is effectively a C API break, and will force
> many thousands of people worldwide to spend many hours wrangling with
> subinterpreter support.

For the record, that's not my reading of the PEP.  PEP 554 doesn't
mandate that C extensions be rewritten to be subinterpreter-compliant.
It makes that an ideal, not an obligation.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RFLPXB4ZD6SA5PS36ZJT6PTMOR6IGNUP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Antoine Pitrou]

On Tue, 21 Apr 2020 18:27:41 +1200
Greg Ewing  wrote:
> On 21/04/20 10:23 am, Eric Snow wrote:
> > with the current spec channels get automatically closed
> > sooner, effectively as soon as all wrapping objects *that were used*
> > are garbage collected (or released).  
> 
> Maybe I'm missing something, but just because an object
> hasn't been used *yet* doesn't mean it isn't going to
> be used in the future, so isn't this wildly wrong?

That's my concern indeed.  An interpreter may be willing to wait for
incoming data in the future, without needing it immediately.

(that incoming data may even represent something very trivial, such as a
request to terminate itself)

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SJ6ORBTNZ7R62QTOZF6UITLYPWZOVRQ7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Antoine Pitrou]

On Tue, 21 Apr 2020 19:45:02 +1200
Greg Ewing  wrote:
> On 21/04/20 11:24 am, Edwin Zimmerman wrote:
> > Socket connections could be passed off from the main interpreter to
> > sub-interpreters for concurrent processing that simply isn't possible
> > with the global GIL  
> 
> I would need convincing that the processing involved things
> that are truly held up by the GIL, rather than things like
> blocking I/O calls that can release the GIL.

In many realistic situations, it will be a mix of both.  For example,
parsing HTTP headers isn't trivial and will consume CPU time with the
GIL taken, if written in pure Python.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FZID7JKTABMVUWS4KV6JFPMNXSZPGQZO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Ronald Oussoren via Python-Dev]

> On 21 Apr 2020, at 03:21, Eric Snow  wrote:
> 
[…]
> On Mon, Apr 20, 2020 at 4:30 PM Nathaniel Smith  wrote:
> 
[…]

>> 
>> But notice that this means that no-one can use subinterpreters at all,
>> until all of their C extensions have had significant reworks to use
>> the new API, which will take years and tons of work -- it's similar to
>> the Python 3 transition. Many libraries will never make the jump.
> 
> Again, that is a grand statement that makes things sound much worse
> than they really are.  I expect very very few extensions will need
> "significant reworks".  Adding PEP 489 support will not take much
> effort, on the order of minutes.  Dealing with process-global state
> will depend on how much, if any.
> 
> Honest question: how many C extensions have process-global state that
> will cause problems under subinterpreters?  In other words, how many
> already break in mod_wsgi?

Fully supporting sub-interpreters in PyObjC will likely be a lot of work, 
mostly due to being able to subclass Objective-C classes from Python.  With 
sub-interpreters a Python script in an interpreter could see an Objective-C 
class in a different sub-interpreter.   The current PyObjC architecture assumes 
that there’s exactly one (sub-)interpreter, that’s probably fixable but is far 
from trivial.

With the current API it might not even be possible to add sub-interpreter 
support (although I write this without having read the PEP). As far as I 
understand proper support for subinterpreters also requires moving away from 
static type definitions to avoid sharing objects between interpreters (that is, 
use the PyType_FromSpec to build types). At first glance this API does not 
support everything I do in PyObjC (fun with metaclasses, in C code).

BTW. I don’t have a an opinion on the PEP itself at this time, mostly because 
it doesn’t match my current use cases.

Ronald

—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IK3SSUVHZ3B6G77X6PNYDXMA42TFD23B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Antoine Pitrou]

On Mon, 20 Apr 2020 19:21:21 -0600
Eric Snow  wrote:
> Honest question: how many C extensions have process-global state that
> will cause problems under subinterpreters?  In other words, how many
> already break in mod_wsgi?

A slightly tricky question is what happens if a PyObject survives
longer than the subinterpreter that created it.

For example, in PyArrow, we allow passing a Python buffer-like object
as a C++ buffer to C++ APIs.  The C++ buffer could conceivably be kept
around by C++ code for some time.  When the C++ buffer is destroyed,
Py_DECREF() is called on the Python object (I figure that we would have
to switch to the future interpreter-aware PyGILState API -- when will
it be available?).  But what happens if the interpreter which created
the Python object is already destroyed at this point?

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4OW3D2ADPTJCEN5MALP3VKDBSOAACABI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Greg Ewing]

On 21/04/20 8:34 pm, Ronald Oussoren via Python-Dev wrote:

As far as I understand proper support for subinterpreters also requires
moving away from static type definition to avoid sharing objects > between 
interpreters (that is, use the PyType_FromSpec to build types).


Which means updating every extension module that defines its
own Python types and was written before PyType_FromSpec existed.
I expect there is a huge number of those.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BRDRFJFCCJAD6TIVYZMZGE7LDBDAIG3M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Victor Stinner]

Hi,

Le sam. 18 avr. 2020 à 19:16, Antoine Pitrou  a écrit :
> Mostly, I hope that by making the
> subinterpreters functionality available to pure Python programmers
> (while it was formally an advanced and arcane part of the C API), we
> will spur of bunch of interesting third-party experimentations,
> including possibilities that we on python-dev have not thought about.
> (...)
> * I think the module should indeed be provisional.  Experimentation may
>   discover warts that call for a change in the API or semantics.  Let's
>   not prevent ourselves from fixing those issues.

Would it make sense to start by adding the module as a private
"_subinterpreters" module but document it? The "_" prefix would be a
reminder that "hey! it's experimental, there is a no backward
compatibility warranty there".

We can also add a big warning in the documentation.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MYRGBV7PIIXHA3V7JZMZCHYYPEKPRHF5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [RELEASE] Python 2.7.18, the end of an era

[Larry Hastings]

On 4/20/20 8:06 AM, Benjamin Peterson wrote:

I'm eudaemonic to announce the immediate availability of Python 2.7.18. [...] 
Over all those years, CPython's core developers and contributors sedulously 
applied bug fixes to the 2.7 branch, no small task as the Python 2 and 3 
branches diverged.



I'm glad you're enjoying your new thesaurus!


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EP4JR7B3IFEO3GL4U76ALTZE7TYRWDDY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Victor Stinner]

Hi Nathaniel,

Thanks for your very interesting analysis :-)


Le ven. 17 avr. 2020 à 23:12, Nathaniel Smith  a écrit :
> - The asyncio designers (esp. Guido) did a very extensive analysis of
> these libraries' design choices, spoke to the maintainers about what
> they'd learned from hard experience, etc.

asyncio and Twisted could perfectly live outside CPython code base.
asyncio even started as a 3rd party project named Tulip for Python
3.3!

Subinterpreters are different: the implementation rely a lot on
CPython internals. Developing it outside CPython would be way harder
if not impossible.

For example, Eric had to add PyThreadState.id and
PyInterpreterState.id members. He made many changes in
Py_NewInterpreter() and Py_EndInterpreter() which are part of Python
internals. Moreover, there are many large refactoring changes which
are required to properly implement subinterpreters, especially to
better isolate each others: _PyRuntimeState structure to better
identify shared globals, PyConfig API (PEP 587), move more structures
per-interpreter (GC state, warnings state, pending calls, etc.).


> - Even today, the limitations imposed by the stdlib release cycle
> still add substantial difficulty to maintaining asyncio

From what I understood, asyncio has a very large API since it has tons
of features (subprocesses, transports and protocols, streams, futures
and tasks, etc.) and requires a lot of glue code to provide the same
API on all platforms which is really hard. I understood that the
subinterpreters module is very light: just a few functions to provide
"channels" and that's it (in short).

$ ./python
Python 3.9.0a5+ (heads/master:6a9e80a931, Apr 21 2020, 02:28:15)
>>> import _xxsubinterpreters
>>> len(dir(_xxsubinterpreters))
30

>>> import asyncio
>>> len(dir(asyncio))
122

>>> dir(_xxsubinterpreters)
['ChannelClosedError', 'ChannelEmptyError', 'ChannelError',
'ChannelID', 'ChannelNotEmptyError', 'ChannelNotFoundError',
'InterpreterID', 'RunFailedError', '__doc__', '__file__',
'__loader__', '__name__', '__package__', '__spec__', '_channel_id',
'channel_close', 'channel_create', 'channel_destroy',
'channel_list_all', 'channel_recv', 'channel_release', 'channel_send',
'create', 'destroy', 'get_current', 'get_main', 'is_running',
'is_shareable', 'list_all', 'run_string']


> OTOH, AFAICT the new concurrency model in PEP 554 has never actually
> been used, and it isn't even clear whether it's useful at all.

Antoine (BDFL-delegate of the PEP) wants to mark the PEP as
provisional: it means that we *can* still remove the whole module
later if we decide that it's not worth it.

The same was done with asyncio. By the way, Antoine was also the
BDFL-delegate of the heavy PEP 3156 (asyncio) ;-)


> Designing useful concurrency models is *stupidly* hard. And on top of
> that, it requires major reworks of the interpreter internals +
> disrupts the existing C extension module ecosystem -- which is very
> different from asyncio, where folks who didn't use it could just
> ignore it.

A lot of work done to isolate subinterpreters also helps to "fix"
CPython internals. For example, converting C extension modules to
multiphase initialization (PEP 489) and add a module state helps to
destroy all module objects at exit and also helps to break reference
cycles in the garbage collector.

It helps to fix this issue created in 2007:
"Py_Finalize() doesn't clear all Python objects at exit"
https://bugs.python.org/issue1635741

It also enhances the implementation of the "embed Python" use case:
"leak" less objects at exit, better cleanup things, etc.

That's why I'm supporter of the overall project: even if
subinterpreters never take off, the implementation will make CPython
better ;-)

FYI it even helps indirectly my project to clean the C API ;-)

PEP 554 implementation looks quite small: 2 800 lines of C code
(Modules/_xxsubinterpretersmodule.c and Objects/interpreteridobject.c)
and 2 100 lines of tests (Lib/test/test__xxsubinterpreters.py).
Compare it to asyncio: 13 000 lines (Lib/asyncio/*.py) of Python and
20 800 lines of tests (Lib/test/test_asyncio/*.py).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LUV4AHNVK4LX2FPUTVQUEU7GYZ5HPDVY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Eric Snow]

On Tue, Apr 21, 2020 at 2:09 AM Antoine Pitrou  wrote:
>
> On Mon, 20 Apr 2020 15:30:37 -0700
> Nathaniel Smith  wrote:
> >
> > tl;dr: accepting PEP 554 is effectively a C API break, and will force
> > many thousands of people worldwide to spend many hours wrangling with
> > subinterpreter support.
>
> For the record, that's not my reading of the PEP.  PEP 554 doesn't
> mandate that C extensions be rewritten to be subinterpreter-compliant.
> It makes that an ideal, not an obligation.

They would not have to be rewritten, unless they want to be used in
subinterpreters.  The PEP does not say this explicitly.  Here are the
options for handling non-compliant extension modules:

1. do nothing (extensions may break in ways that are hard for users to
deal with)
2. raise ImportError if an extension without PEP 489 support is
imported in a subinterpreter
3. add a per-interpreter isolated mode (default) that provides the
behavior of #2, where non-isolated gives #1

At this point my plan was go with #2, to avoid giving extension
authors the pain that comes with #1.  If we went with #3, at least
during the provisional phase, it would allow extension authors to try
out their extensions in subinterpreters.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E4456ZIXUK7O2QZ4DJETMKTSFYADFL7B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Victor Stinner]

Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith  a écrit :
> Why do you believe that subinterpreters will have reduced resource
> usage? I assume you're comparing them to subprocesses here.
> Subinterpreters are "shared-nothing"; all code, data, etc. has to be
> duplicated, except for static C code ... which is exactly the same as
> how subprocesses work. So I don't see any theoretical reason why they
> should have reduced resource usage.

As you, I would like to see benchmarks of subinterpreters running in
parallel before saying that "yeah, that approach works to speed up
this specific use case". But I don't think that it should hold the PEP
554. Eric is transparent about his intent and limitations of the
current implementation.

There is no silver bullet for parallelism and concurrency. For
example, multiprocessing uses threads in addition to processes, and
asyncio also uses threads internally to execute blocking code like DNS
resolution.

IMHO it's worth it to explore the subinterpreters approach. I don't
think that it's going to be a major maintenance burden: the code is
already written and tests. The PEP is only about adding a public
Python API.



> tl;dr: accepting PEP 554 is effectively a C API break, and will force
> many thousands of people worldwide to spend many hours wrangling with
> subinterpreter support.

I fail to follow your logic. When the asyncio PEP was approved, I
don't recall that suddenly the whole Python community started to
rewrite all projects to use coroutines everywhere. I tried hard to
replace eventlet with asyncio in OpenStack and I failed because such
migration was a very large project with dubious benefits (people
impacted by eventlet issues were the minority).

When asyncio landed in Python 3.4, a few people started to experiment
it. Some had a bad experience. Some others were excited and put a few
applications in production.

Even today, asyncio didn't replace threads, multiprocessing,
concurrent.futures, etc. There are even competitor projects like
Twisted, trio and curio! (Also eventlet and gevent based on greenlet
which is a different approach). I only started to see very recently
project like httpx which supports both blocking and asynchronous API.

I see a slow adoption of asyncio because asyncio solves very specific
use cases. And that's fine!

I don't expect that everyone will suddenly spend months of work to
rewrite their C code and Python code to be more efficient or fix
issues with subinterpreters, until a critical mass of users proved
that subinterpreters are amazing and way more efficient!

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3MK2NANMOJFHKJWELGI2ZD74K2ZYJAOD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [RELEASE] Python 2.7.18, the end of an era

[Brian Curtin]

On Tue, Apr 21, 2020 at 7:31 AM Larry Hastings  wrote:

> On 4/20/20 8:06 AM, Benjamin Peterson wrote:
>
> I'm eudaemonic to announce the immediate availability of Python 2.7.18. [...] 
> Over all those years, CPython's core developers and contributors sedulously 
> applied bug fixes to the 2.7 branch, no small task as the Python 2 and 3 
> branches diverged.
>
>
> I'm glad you're enjoying your new thesaurus!
>
I'm glad he was able to keep up the ebullience so long. He was jocund about
2.7.0rc2 around this time ten years ago.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JWYWGC7CO7FZP7KVGZO3VHLCTBRGCUWP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Victor Stinner]

__future__ imports only have effects on the parser and compiler.

PEP 554 is mostly a Python module, currently named "_xxsubinterpreters".

Victor

Le mar. 21 avr. 2020 à 15:37, Edwin Zimmerman
 a écrit :
>
> On Tuesday, April 21, 2020 9:20 AM Victor Stinner 
> [mailto:vstin...@python.org] wrote
> > Hi,
> >
> > Le sam. 18 avr. 2020 à 19:16, Antoine Pitrou  a écrit :
> > > Mostly, I hope that by making the
> > > subinterpreters functionality available to pure Python programmers
> > > (while it was formally an advanced and arcane part of the C API), we
> > > will spur of bunch of interesting third-party experimentations,
> > > including possibilities that we on python-dev have not thought about.
> > > (...)
> > > * I think the module should indeed be provisional.  Experimentation may
> > >   discover warts that call for a change in the API or semantics.  Let's
> > >   not prevent ourselves from fixing those issues.
> >
> > Would it make sense to start by adding the module as a private
> > "_subinterpreters" module but document it? The "_" prefix would be a
> > reminder that "hey! it's experimental, there is a no backward
> > compatibility warranty there".
> >
> > We can also add a big warning in the documentation.
> >
> > Victor
>
>  What about requiring "from __future__ import subinterpreters" to use this?
> According to the docs, the purpose of __future__ is "to document when 
> incompatible changes were introduced", and it does seem that this would be an 
> incompatible change for some C extensions.
> --Edwin
>


-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NN4FD6MN5UHMNQDG6QBVVVOSHXLDUQYA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Edwin Zimmerman]

On Tuesday, April 21, 2020 9:20 AM Victor Stinner [mailto:vstin...@python.org] 
wrote
> Hi,
> 
> Le sam. 18 avr. 2020 à 19:16, Antoine Pitrou  a écrit :
> > Mostly, I hope that by making the
> > subinterpreters functionality available to pure Python programmers
> > (while it was formally an advanced and arcane part of the C API), we
> > will spur of bunch of interesting third-party experimentations,
> > including possibilities that we on python-dev have not thought about.
> > (...)
> > * I think the module should indeed be provisional.  Experimentation may
> >   discover warts that call for a change in the API or semantics.  Let's
> >   not prevent ourselves from fixing those issues.
> 
> Would it make sense to start by adding the module as a private
> "_subinterpreters" module but document it? The "_" prefix would be a
> reminder that "hey! it's experimental, there is a no backward
> compatibility warranty there".
> 
> We can also add a big warning in the documentation.
> 
> Victor

 What about requiring "from __future__ import subinterpreters" to use this?
According to the docs, the purpose of __future__ is "to document when 
incompatible changes were introduced", and it does seem that this would be an 
incompatible change for some C extensions.
--Edwin
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N357K6CNGLKWILZYTZUTCIHSS2IEEEXG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Paul Moore]

On Tue, 21 Apr 2020 at 15:31, Eric Snow  wrote:
> Here are the options for handling non-compliant extension modules:
>
> 1. do nothing (extensions may break in ways that are hard for users to
> deal with)
> 2. raise ImportError if an extension without PEP 489 support is
> imported in a subinterpreter
> 3. add a per-interpreter isolated mode (default) that provides the
> behavior of #2, where non-isolated gives #1
>
> At this point my plan was go with #2, to avoid giving extension
> authors the pain that comes with #1.  If we went with #3, at least
> during the provisional phase, it would allow extension authors to try
> out their extensions in subinterpreters.

This seems sensible to me - #2 ensures that we don't get weird bugs
from users trying things that we know they shouldn't be doing. I'm not
sure how much advantage #3 would have, presumably extension authors
wanting to "try out" subinterpreters would at some point have to mark
them as importable as per #2, so why not just do that much and then
see what else needs changing? But I don't know the C API for
subinterpreters, so maybe there are complexities I don't understand,
in which case having #3 probably makes more sense in that context.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KFOJQ3Z7AS3PCCAIOGA2KPGXXJWEYIZK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Eric Snow]

Thanks for explaining that, Ronald.  It sounds like a lot of the
effort would relate to making classes work.  I have some comments
in-line below.

-eric

On Tue, Apr 21, 2020 at 2:34 AM Ronald Oussoren  wrote:
> > On 21 Apr 2020, at 03:21, Eric Snow  wrote:
> > Honest question: how many C extensions have process-global state that
> > will cause problems under subinterpreters?  In other words, how many
> > already break in mod_wsgi?
>
> Fully supporting sub-interpreters in PyObjC will likely be a lot of work, 
> mostly
> due to being able to subclass Objective-C classes from Python.  With sub-
> interpreters a Python script in an interpreter could see an Objective-C class 
> in
> a different sub-interpreter.   The current PyObjC architecture assumes that
> there’s exactly one (sub-)interpreter, that’s probably fixable but is far 
> from trivial.

Are the Objective-C classes immutable?  Are the wrappers stateful at
all?  Without context I'm not clear on how you would be impacted by
operation under subinterpreters (i.e. PEP 554), but it sounds like the
classes do have global state that is in fact interpreter-specific.  I
expect you would also be impacted by subinterpreters not sharing the
GIL but that is a separate matter (see below).

Regardless, I expect there are others in a similar situation.  It
would be good to understand your use case and help with a solution.
Is there a specific example you can point to of code that would be
problematic under subinterpreters?

> With the current API it might not even be possible to add sub-interpreter 
> support

What additional API would be needed?

> (although I write this without having read the PEP).

Currently PEP 554 does not talk about how to make extension modules
compatible with subinterpreters.  That may be worth doing, though it
would definitely have to happen in the docs and (to an extent) the 3.9
"What's New" page.  There is already some discussion on what should be
in those docs (see
https://github.com/ericsnowcurrently/multi-core-python/issues/53).

Note that, until the GIL becomes per-interpreter, sharing objects
isn't a problem.  We were not planning on having a PEP for the
stop-sharing-the-GIL effort, but I'm starting to think that it may be
worth it, to cover the impact on extension modules (e.g. mitigations).

So if you leave out the complications due to not sharing the GIL, the
main problem extension authors face with subinterpreters (exposed by
PEP 554) is when their module has process-global state that breaks
under subinterpreters.  From your description above, it sounds like
you may be in that situation.

> As far as I understand proper support for subinterpreters also requires moving
> away from static type definitions to avoid sharing objects between 
> interpreters
> (that is, use the PyType_FromSpec to build types).

Correct, though that is not technically a problem until we stop sharing the GIL.

> At first glance this API does not support everything I do in PyObjC (fun with 
> metaclasses, in C code).

What specific additions/changes would you need?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UIX7FM2UJUL2UIIBNOBYNP42O6RCSJME/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Eric Snow]

On Tue, Apr 21, 2020 at 3:10 AM Antoine Pitrou  wrote:
> On Mon, 20 Apr 2020 19:21:21 -0600
> Eric Snow  wrote:
> > Honest question: how many C extensions have process-global state that
> > will cause problems under subinterpreters?  In other words, how many
> > already break in mod_wsgi?
>
> A slightly tricky question is what happens if a PyObject survives
> longer than the subinterpreter that created it.
>
> For example, in PyArrow, we allow passing a Python buffer-like object
> as a C++ buffer to C++ APIs.  The C++ buffer could conceivably be kept
> around by C++ code for some time.  When the C++ buffer is destroyed,
> Py_DECREF() is called on the Python object (I figure that we would have
> to switch to the future interpreter-aware PyGILState API -- when will
> it be available?).  But what happens if the interpreter which created
> the Python object is already destroyed at this point?

Good question (and example).  I'm not familiar enough with the GC
machinery to have a good answer.  I do know  that GC is
per-interpreter now, but the allocators are still global.  I'm not
sure how object lifetime is *currently* tied to interpreter lifetime.
I'd expect that, at the point we have per-interpreter allocators, that
all objects in that interpreter would be destroyed when the
interpreter is destroyed.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7DJCPVP7UA66WMEF7AWKAEWUC5N37WZN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Eric Snow]

On Tue, Apr 21, 2020 at 4:11 AM Greg Ewing  wrote:
> On 21/04/20 8:34 pm, Ronald Oussoren via Python-Dev wrote:
> > As far as I understand proper support for subinterpreters also requires
> > moving away from static type definition to avoid sharing objects > between 
> > interpreters (that is, use the PyType_FromSpec to build types).
>
> Which means updating every extension module that defines its
> own Python types and was written before PyType_FromSpec existed.
> I expect there is a huge number of those.

Correct.  Before we make the GIL per-interpreter, we will need to find
a solution for things like static types which minimizes the impact on
extension authors.  Note that PyType_FromSpec does not have to be
involved.  It is part of the Stable ABI effort in PEP 384, though it
does help with subinterpreters.  I expect there are other solutions
which will not involve as much churn, though I know folks who would
want everyone to use heap types regardless. :)

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6PWGTPZIFSHVACVTZ6YFA7MYY7L22G4O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Eric Snow]

On Tue, Apr 21, 2020 at 8:54 AM Paul Moore  wrote:
>
> On Tue, 21 Apr 2020 at 15:31, Eric Snow  wrote:
> > Here are the options for handling non-compliant extension modules:
> >
> > 1. do nothing (extensions may break in ways that are hard for users to
> > deal with)
> > 2. raise ImportError if an extension without PEP 489 support is
> > imported in a subinterpreter
> > 3. add a per-interpreter isolated mode (default) that provides the
> > behavior of #2, where non-isolated gives #1
> >
> > At this point my plan was go with #2, to avoid giving extension
> > authors the pain that comes with #1.  If we went with #3, at least
> > during the provisional phase, it would allow extension authors to try
> > out their extensions in subinterpreters.
>
> This seems sensible to me - #2 ensures that we don't get weird bugs
> from users trying things that we know they shouldn't be doing. I'm not
> sure how much advantage #3 would have, presumably extension authors
> wanting to "try out" subinterpreters would at some point have to mark
> them as importable as per #2, so why not just do that much and then
> see what else needs changing?

Good point.  I suppose the main difference is that they (and their
users) could try the extension under subinterpreters without needing a
separate build.

> But I don't know the C API for
> subinterpreters, so maybe there are complexities I don't understand,
> in which case having #3 probably makes more sense in that context.

There aren't any extra complexities.  Per PEP 489 [1], in a module's
init function you call PyModuleDef_Init() on its PyModuleDef and the
return it.  The import machinery does the rest.  The subinterpreter
C-API is not involved.

-eric

[1] https://www.python.org/dev/peps/pep-0489/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ACYSACREHESNOUIJLXZLRZUBORZ4Q7GS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Eric Snow]

On Tue, Apr 21, 2020 at 2:18 AM Antoine Pitrou  wrote:
> On Tue, 21 Apr 2020 18:27:41 +1200
> Greg Ewing  wrote:
> > On 21/04/20 10:23 am, Eric Snow wrote:
> > > with the current spec channels get automatically closed
> > > sooner, effectively as soon as all wrapping objects *that were used*
> > > are garbage collected (or released).
> >
> > Maybe I'm missing something, but just because an object
> > hasn't been used *yet* doesn't mean it isn't going to
> > be used in the future, so isn't this wildly wrong?
>
> That's my concern indeed.  An interpreter may be willing to wait for
> incoming data in the future, without needing it immediately.
>
> (that incoming data may even represent something very trivial, such as a
> request to terminate itself)

Yeah, I had that same realization yesterday and it didn't change after
sleeping on it.  I suppose the only question I have left is if there
is value to users in knowing which interpreters have *used* a
particular channel.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LN5RLD7FCTLVFLNJZWW6TNCTDETMHYBQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Eric Snow]

On Tue, Apr 21, 2020 at 1:39 AM Greg Ewing  wrote:
> I don't get this whole business of channels being associated
> with interpreters, or why there needs to be a distinction
> between release() and close().
>
> To my mind, a channel reference should be like a file
> descriptor for a pipe. When you've finished with it, you
> close() it. When the last reference to a channel is
> closed or garbage collected, the channel disappears.
> Why make it any more complicated than that?

You've mostly described what the PEP proposes: when all objects
wrapping a channel in an interpreter are destroyed, that channel is
automatically released.  When all interpreters have released the
channel then it is automatically closed.

The main difference is that the PEP also provides an way to explicitly
release or close a channel.  Providing just "close()" would mean one
interpreter could stomp on all other interpreters' use of a channel.
Working around that would require clunky coordination (likely through
other channels).  The alternative ("release()") is much simpler.

> You seem to be worried about channels getting leaked if
> someone forgets to close them. But it's just the same
> for files and pipes, and nobody seems to worry about
> that.

Fair enough. :)

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D6ML5MST6ZJJP2WYGYQWOLLR6GGI6MMZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Eric Snow]

On Tue, Apr 21, 2020 at 7:25 AM Victor Stinner  wrote:
> Would it make sense to start by adding the module as a private
> "_subinterpreters" module but document it? The "_" prefix would be a
> reminder that "hey! it's experimental, there is a no backward
> compatibility warranty there".

I would expect a leading underscore to be confusing (as well as
conflicting with the name of the low-level module).  if we did
anything then it would probably make more sense to name the module
something like "interpreters_experimental".  However, I'm not sure
that offers that much benefit.

> We can also add a big warning in the documentation.

We will mark it "provisional" in the docs, which I expect will include
info on what that means and why it is provisional.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LB7ACPFAT3QU5YUPIFWH4NHB7YYVJXQG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Sebastian Berg]

On Tue, 2020-04-21 at 16:21 +0200, Victor Stinner wrote:
> Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith  a écrit
> :

> 
> 
> > tl;dr: accepting PEP 554 is effectively a C API break, and will
> > force
> > many thousands of people worldwide to spend many hours wrangling
> > with
> > subinterpreter support.
> 
> I fail to follow your logic. When the asyncio PEP was approved, I
> don't recall that suddenly the whole Python community started to
> rewrite all projects to use coroutines everywhere. I tried hard to
> replace eventlet with asyncio in OpenStack and I failed because such
> migration was a very large project with dubious benefits (people
> impacted by eventlet issues were the minority).

Sure, but this is very different. You can still use NumPy in a project
using asyncio. You are _not_ able to use NumPy in a project using
subinterpreters.
Right now, I have to say as soon as the first bug report asking for
this is opened and tells me: But see PEP 554 you should support it! I
would be tempted to put on the NumPy Roadmap/Vision that no current
core dev will put serious efforts into subinterpreters. Someone is
bound to be mad.
Basically, if someone wants it in NumPy, I personally may expect them
to be prepared to invest a year worth of good dev time [1]. Maybe that
is pessimistic, but your guess is as good as mine. At normal dev-pace
it will be at least a few years of incremental changes before NumPy
might be ready (how long did it take Python?)?

The PEP links to NumPy bugs, I am not sure that we ever fixed a single
one. Even if, the remaining ones are much larger and deeper. As of now,
the NumPy public API has to be changed to even start supporting
subinterpreters as far as I aware [2]. This is because right now we
sometimes need to grab the GIL (raise errors) in functions that are not
passed GIL state.


This all is not to say that this PEP itself doesn't seem harmless. But
the _expectation_ that subinterpreters should be first class citizens
will be a real and severe transition burden. And if it does not, the
current text of the PEP gives me, as someone naive about
subinterpreters, very few reasons why I should put in that effort or
reasons to make me believe that it actually is not as bad a transition
as it seems.

Right now, I would simply refuse to spend time on it. But as Nathaniel
said, it may be worse if I did not refuse and in the end only a handful
of users get anything out of my work: The time is much better spend
elsewhere. And you, i.e. CPython will spend your "please fix your C-
extension" chips on subinterpreters. Maybe that is the only thing on
the agenda, but if it is not, it could push other things away.

Reading the PEP, it is fuzzy on the promises (the most concrete I
remember is that it may be good for security relevant reasons), which
is fine, because the goal is "experimentation" more than use?

So if its more about "experimentation", then I have to ask, whether:

1. The PEP can state that more obviously, it wants to be
provisionally/experimentally accept? So maybe it should even say that
that extension modules are not (really) encouraged to transition unless
they feel a significant portion of their users will gain.

2. The point about developing it outside of the Python standard lib
should be considered more seriously. I do not know if that can be done,
but C-API additions/changes/tweaks seem a bit orthogonal to the python
exposure? So maybe it actually is possible?

As far as I can tell, nobody can or _should_ expect subinterpreters to
actually run most general python code for many years. Yes, its a
chicken-and-egg problem, unless users start to use subinterpreters
successfully, C-extensions should probably not even worry to
transition.
This PEP wants to break the chicken-and-egg problem to have a start,
but as of now, as far as I can tell, it *must not* promise that it will
ever work out.

So, I cannot judge the sentiment or subinterpreters. But it may be good
to make it *painfully* clear what you expect from a project like NumPy
in the next few years. Alternatively, make it painfully clear that you
possibly even discourage us from spending time on it now, if its not
straight forward. Those using this module are on their own for many
years, probably even after success is proven.

Best,

Sebastian


[1] As of now, the way I see it is that I could not even make NumPy
(and probably many C extensions) work, because I doubt that the limited
API has been exercised enough [2] and I am pretty sure it has holes.
Also the PEP about passing module state around to store globals
efficiently seems necessary, and is not in yet? (Again, trust: I have
to trust you that e.g. what you do to make argument parsing not have
overhead in argument clinic will be something that I can use for
similar purposes within NumPy)

[2]  I hope that we will do (many) these changes for other reasons
within a year or so, but they go deep into code barely touched in a
decade. Realistically, even after the straigh

[Python-Dev] Re: PEP 554 comments

[Antoine Pitrou]

On Tue, 21 Apr 2020 09:36:22 -0600
Eric Snow  wrote:

> On Tue, Apr 21, 2020 at 2:18 AM Antoine Pitrou  wrote:
> > On Tue, 21 Apr 2020 18:27:41 +1200
> > Greg Ewing  wrote:  
> > > On 21/04/20 10:23 am, Eric Snow wrote:  
> > > > with the current spec channels get automatically closed
> > > > sooner, effectively as soon as all wrapping objects *that were used*
> > > > are garbage collected (or released).  
> > >
> > > Maybe I'm missing something, but just because an object
> > > hasn't been used *yet* doesn't mean it isn't going to
> > > be used in the future, so isn't this wildly wrong?  
> >
> > That's my concern indeed.  An interpreter may be willing to wait for
> > incoming data in the future, without needing it immediately.
> >
> > (that incoming data may even represent something very trivial, such as a
> > request to terminate itself)  
> 
> Yeah, I had that same realization yesterday and it didn't change after
> sleeping on it.  I suppose the only question I have left is if there
> is value to users in knowing which interpreters have *used* a
> particular channel.

I don't think so :-)

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BP2UHLS5MNX6ZZYVHVXNZQ4W62GUW62M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Comments on PEP 554 (Multiple Interpreters in the Stdlib)

[Mark Shannon]

Hi,

I'm generally in favour of PEP 554, but I don't think it is ready to be 
accepted in its current form.


My main objection is that without per-subinterpeter GILs (SILs?) PEP 554 
provides no value over threading or multi-processing.
Multi-processing provides true parallelism and threads provide shared 
memory concurrency.


If per-subinterpeter GILs are possible then, and only then, 
sub-interpreters will provide true parallelism and (limited) shared 
memory concurrency.


The problem is that we don't know whether we can implement 
per-subinterpeter GILs without too large a negative performance impact.

I think we can, but we can't say so for certain.

So, IMO, we should not accept PEP 554 until we know for sure that 
per-subinterpeter GILs can be implemented efficiently.




Detailed critique
-

I don't see how `list_all()` can be both safe and accurate. The Java 
equivalent makes no guarantees of accuracy.
Attempting to lock the list is likely to lead to deadlock and not 
locking it will lead to races; potentially dangerous ones.

I think it would be best to drop this.

`list_all_channels()`. See `list_all()` above.

`.destroy()` is either misleading or unsafe.
What does this do?

>>> is.destroy()
>>> is.run()

If `run()` raises an exception then the interpreter must exist. Rename 
to `close()` perhaps?


`Channel.interpreters` see `list_all()` and `list_all_channels()` above.

How does `is_shareable()` work? Are you proposing some mechanism to 
transfer an object from one sub-interpreter to another? How would that 
work? If objects are not shared but serialized, why not use marshal or 
pickle instead of inventing a third serialization protocol?


It would be clearer if channels only dealt with simple, contiguous 
binary data. As it stands the PEP doesn't state what form the received 
object will take.
Once channels supporting the transfer of bytes objects work, then it is 
simple to pass more complex objects using pickle or marshal.


Channels need a more detailed description of their lifespan. Ideally a 
state machine.

For example:
How does an interpreter detach from the receiving end of a channel that 
is never empty?
What happens if an interpreter deletes the last reference to a non-empty 
channel? On the receiving end, or on the sending end?


Isn't the lack of buffering in channels a recipe for deadlocks?

What is the mechanism for reliably copying exceptions from one 
sub-interpreter to another in the `run()` method? If `run()` can raise 
an exception, why not let it return values?



Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZSE2G37E24YYLNMQKOQSBM46F7KLAOZF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Eric Snow]

On Tue, Apr 21, 2020 at 10:33 AM Antoine Pitrou  wrote:
> On Tue, 21 Apr 2020 09:36:22 -0600
> Eric Snow  wrote:
> > Yeah, I had that same realization yesterday and it didn't change after
> > sleeping on it.  I suppose the only question I have left is if there
> > is value to users in knowing which interpreters have *used* a
> > particular channel.
>
> I don't think so :-)

Yeah, as with so much with this PEP, if it proves desirable then we
can add it later.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MRHMNPSWONFIE5LGOCFL3DE4KFX6NELW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Petr Viktorin]

On 2020-04-21 11:01, Antoine Pitrou wrote:

On Mon, 20 Apr 2020 19:21:21 -0600
Eric Snow  wrote:

Honest question: how many C extensions have process-global state that
will cause problems under subinterpreters?  In other words, how many
already break in mod_wsgi?


A slightly tricky question is what happens if a PyObject survives
longer than the subinterpreter that created it.

For example, in PyArrow, we allow passing a Python buffer-like object
as a C++ buffer to C++ APIs.  The C++ buffer could conceivably be kept
around by C++ code for some time.  When the C++ buffer is destroyed,
Py_DECREF() is called on the Python object (I figure that we would have
to switch to the future interpreter-aware PyGILState API -- when will
it be available?). 




But what happens if the interpreter which created
the Python object is already destroyed at this point?


That's a good question. What happens today if someone calls Py_Finalize 
while the buffer is still around?


I don't think you need to treat *sub*interpreters specially.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AVJO43PRTX4UCMLI2PNHMZWWBEGZS353/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Sebastian Berg]

On Tue, 2020-04-21 at 11:21 -0500, Sebastian Berg wrote:
> On Tue, 2020-04-21 at 16:21 +0200, Victor Stinner wrote:
> > Le mar. 21 avr. 2020 à 00:50, Nathaniel Smith  a
> > écrit
> > :
> 

> As far as I can tell, nobody can or _should_ expect subinterpreters
> to
> actually run most general python code for many years. Yes, its a
> chicken-and-egg problem, unless users start to use subinterpreters
> successfully, C-extensions should probably not even worry to
> transition.
> This PEP wants to break the chicken-and-egg problem to have a start,
> but as of now, as far as I can tell, it *must not* promise that it
> will
> ever work out.
> 
> So, I cannot judge the sentiment or subinterpreters. But it may be
> good
> to make it *painfully* clear what you expect from a project like
> NumPy

Maybe one of the frustrating points about this criticism is that it
does not belong in this PEP. And that is actually true! I
wholeheartedly agree that it doesn't really belong in this PEP itself.

*But* the existence of a document detailing the "state and vision for
subinterpreters" that includes these points is probably a prerequisite
for this PEP. And this document must be linked prominently from the
PEP.

So the suggestion should maybe not be to discuss it in the PEP, but to
to write it either in the documentation on subinterpreters or as an
informational PEP. Maybe such document already exists, but then it is
not linked prominently enough probably.

- Sebastian


> in the next few years. Alternatively, make it painfully clear that
> you
> possibly even discourage us from spending time on it now, if its not
> straight forward. Those using this module are on their own for many
> years, probably even after success is proven.
> 
> Best,
> 
> Sebastian
> 
> 
> [1] As of now, the way I see it is that I could not even make NumPy
> (and probably many C extensions) work, because I doubt that the
> limited
> API has been exercised enough [2] and I am pretty sure it has holes.
> Also the PEP about passing module state around to store globals
> efficiently seems necessary, and is not in yet? (Again, trust: I have
> to trust you that e.g. what you do to make argument parsing not have
> overhead in argument clinic will be something that I can use for
> similar purposes within NumPy)
> 
> [2]  I hope that we will do (many) these changes for other reasons
> within a year or so, but they go deep into code barely touched in a
> decade. Realistically, even after the straight forward changes (such
> as
> using the new PEPs for module initialization), these may take up an
> additional few months of dev time (sure, get someone very good or
> does
> nothing else, they can do it much quicker maybe).
> So yes, from the perspective of a complex C-extension, this is
> probably
> very comparable to the 2to3 change (it happened largely before my
> time
> though).
> 
> [3] E.g. I think I want an ExtensionMetaClass, a bit similar as an
> ABC,
> but I would prefer to store the data in a true C-slot fashion. The
> limited API cannot do MetaClasses correctly as far as I could tell
> and
> IIRC is likely even a bit buggy.
> Are ExtensionMetaClasses crazy? Maybe, but PySide does it too (and as
> far as I can tell, they basically get away with it by a bit of
> hacking
> and relying on Python implementation details.
> 
> 
> 
> > When asyncio landed in Python 3.4, a few people started to
> > experiment
> > it. Some had a bad experience. Some others were excited and put a
> > few
> > applications in production.
> > 
> > Even today, asyncio didn't replace threads, multiprocessing,
> > concurrent.futures, etc. There are even competitor projects like
> > Twisted, trio and curio! (Also eventlet and gevent based on
> > greenlet
> > which is a different approach). I only started to see very recently
> > project like httpx which supports both blocking and asynchronous
> > API.
> > 
> > I see a slow adoption of asyncio because asyncio solves very
> > specific
> > use cases. And that's fine!
> > 
> > I don't expect that everyone will suddenly spend months of work to
> > rewrite their C code and Python code to be more efficient or fix
> > issues with subinterpreters, until a critical mass of users proved
> > that subinterpreters are amazing and way more efficient!
> > 
> > Victor
> > -- 
> > Night gathers, and now my watch begins. It shall not end until my
> > death.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at 
> > https://mail.python.org/archives/list/python-dev@python.org/message/3MK2NANMOJFHKJWELGI2ZD74K2ZYJAOD/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.or

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Antoine Pitrou]

On Tue, 21 Apr 2020 18:46:04 +0200
Petr Viktorin  wrote:
> On 2020-04-21 11:01, Antoine Pitrou wrote:
> > On Mon, 20 Apr 2020 19:21:21 -0600
> > Eric Snow  wrote:  
> >> Honest question: how many C extensions have process-global state that
> >> will cause problems under subinterpreters?  In other words, how many
> >> already break in mod_wsgi?  
> > 
> > A slightly tricky question is what happens if a PyObject survives
> > longer than the subinterpreter that created it.
> > 
> > For example, in PyArrow, we allow passing a Python buffer-like object
> > as a C++ buffer to C++ APIs.  The C++ buffer could conceivably be kept
> > around by C++ code for some time.  When the C++ buffer is destroyed,
> > Py_DECREF() is called on the Python object (I figure that we would have
> > to switch to the future interpreter-aware PyGILState API -- when will
> > it be available?).   
> 
> 
> > But what happens if the interpreter which created
> > the Python object is already destroyed at this point?  
> 
> That's a good question. What happens today if someone calls Py_Finalize 
> while the buffer is still around?
> 
> I don't think you need to treat *sub*interpreters specially.

The difference is that subinterpreters may have a shorter lifetime than
the main interpreter.  Also, we're currently using a global memory
allocator, so simple operations may work until the process dies.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UWWO3RS3UOYCZXVFORGTJAEXXOCUQQXD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Gregory P. Smith]

On Tue, Apr 21, 2020 at 10:49 AM Antoine Pitrou  wrote:

> On Tue, 21 Apr 2020 18:46:04 +0200
> Petr Viktorin  wrote:
> > On 2020-04-21 11:01, Antoine Pitrou wrote:
> > > On Mon, 20 Apr 2020 19:21:21 -0600
> > > Eric Snow  wrote:
> > >> Honest question: how many C extensions have process-global state that
> > >> will cause problems under subinterpreters?  In other words, how many
> > >> already break in mod_wsgi?
> > >
> > > A slightly tricky question is what happens if a PyObject survives
> > > longer than the subinterpreter that created it.
> > >
> > > For example, in PyArrow, we allow passing a Python buffer-like object
> > > as a C++ buffer to C++ APIs.  The C++ buffer could conceivably be kept
> > > around by C++ code for some time.  When the C++ buffer is destroyed,
> > > Py_DECREF() is called on the Python object (I figure that we would have
> > > to switch to the future interpreter-aware PyGILState API -- when will
> > > it be available?).
>

In this scenario given that the PyObject owning the buffer passed to C++ by
PyArrow was Py_INCREF'd, an option during Py_Finalize is to not release
anything who's refcount never made it to 0.  Which also implies leaving
enough of the interpreter state around so that Py_DECREF could be called
and trigger the free().

Basically (sub)interpreter finalization should have the right to return
failure.

Everything ever allocated by an interpreter holds a reference (implied
today) to its associated interpreter state.

-gps


> >
> >
> > > But what happens if the interpreter which created
> > > the Python object is already destroyed at this point?
> >
> > That's a good question. What happens today if someone calls Py_Finalize
> > while the buffer is still around?
> >
> > I don't think you need to treat *sub*interpreters specially.
>
> The difference is that subinterpreters may have a shorter lifetime than
> the main interpreter.  Also, we're currently using a global memory
> allocator, so simple operations may work until the process dies.


> Regards
>
> Antoine.
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UWWO3RS3UOYCZXVFORGTJAEXXOCUQQXD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LJJVHHK7OG6ZRISCT6G2NWBBIYDGFHVB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Antoine Pitrou]

On Tue, 21 Apr 2020 12:05:28 -0700
"Gregory P. Smith"  wrote:
> On Tue, Apr 21, 2020 at 10:49 AM Antoine Pitrou  wrote:
> 
> > On Tue, 21 Apr 2020 18:46:04 +0200
> > Petr Viktorin  wrote:  
> > > On 2020-04-21 11:01, Antoine Pitrou wrote:  
> > > > On Mon, 20 Apr 2020 19:21:21 -0600
> > > > Eric Snow  wrote:  
> > > >> Honest question: how many C extensions have process-global state that
> > > >> will cause problems under subinterpreters?  In other words, how many
> > > >> already break in mod_wsgi?  
> > > >
> > > > A slightly tricky question is what happens if a PyObject survives
> > > > longer than the subinterpreter that created it.
> > > >
> > > > For example, in PyArrow, we allow passing a Python buffer-like object
> > > > as a C++ buffer to C++ APIs.  The C++ buffer could conceivably be kept
> > > > around by C++ code for some time.  When the C++ buffer is destroyed,
> > > > Py_DECREF() is called on the Python object (I figure that we would have
> > > > to switch to the future interpreter-aware PyGILState API -- when will
> > > > it be available?).  
> >  
> 
> In this scenario given that the PyObject owning the buffer passed to C++ by
> PyArrow was Py_INCREF'd, an option during Py_Finalize is to not release
> anything who's refcount never made it to 0.  Which also implies leaving
> enough of the interpreter state around so that Py_DECREF could be called
> and trigger the free().

That's assuming a trivial destructor.  Of course, that would cater for
most concrete use cases.

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CJFM5WBERJU4NLW6QJ7VISGFVBCQ3HMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Carl Meyer]

On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum  wrote:
>
> Note that, while there is indeed a docs page about 2to3, the only docs for 
> lib2to3 in the standard library reference are a link to the source code and a 
> single "Note: The lib2to3 API should be considered unstable and may change 
> drastically in the future."
>
> Fortunately,  in order to support the 2to3 application, lib2to3 doesn't need 
> to change, because the syntax of Python 2 is no longer changing. :-) Choosing 
> to remove 2to3 is an independent decision. And lib2to3 does not depend in any 
> way on the old parser module. (It doesn't even use the standard tokenize 
> module, but incorporates its own version that is slightly tweaked to support 
> Python 2.)

Indeed! Thanks for clarifying, I now recall that I already knew it
doesn't, but forgot.

The docs page for 2to3 does currently say "lib2to3 could also be
adapted to custom applications in which Python code needs to be edited
automatically." Perhaps at least this sentence should be removed, and
maybe also replaced with a clearer note that lib2to3 not only has an
unstable API, but also should not necessarily be expected to continue
to parse future Python versions, and thus building tools on top of it
should be discouraged rather than recommended. (Maybe even use the
word "deprecated.") Happy to submit a PR for this if you agree it's
warranted.

It still seems to me that it wouldn't hurt for PEP 617 itself to also
mention this shift in lib2to3's effective status (from "available but
no API stability guarantee" to "probably will not parse future Python
versions") as one of its indirect effects.

> You've mentioned a few different tools that already use different 
> technologies: LibCST depends on parso which has a fork of pgen2, lib2to3 
> which has the original pgen2. I wonder if this would be an opportunity to 
> move such parsing support out of the standard library completely. There are 
> already two versions of pegen, but neither is in the standard library: there 
> is the original pegen repo which is where things started, and there is a fork 
> of that code in the CPython Tools directory (not yet in the upstream repo, 
> but see PR 19503).
>
> The pegen tool has two generators, one generating C code and one generating 
> Python code. I think that the C generator is really only relevant for CPython 
> itself: it relies on the builtin tokenizer (the one written in C, not the 
> stdlib tokenize.py) and the generated C code depends on many internal APIs. 
> In fact the C generator in the original pegen repo doesn't work with Python 
> 3.9 because those internal APIs are no longer exported. (It also doesn't work 
> with Python 3.7 or older because it makes critical use of the walrus 
> operator. :-) Also, once we started getting serious about replacing the old 
> parser, we worked exclusively on the C generator in the CPython Tools 
> directory, so the version in the original pegen repo is lagging quite a bit 
> behind (is is the Python grammar in that repo). But as I said you're not 
> gonna need it.
>
> On the other hand, the Python generator is designed to be flexible, and while 
> it defaults to using the stdlib tokenize.py tokenizer, you can easily hook up 
> your own. Putting this version in the stdlib would be a mistake, because the 
> code is pretty immature; it is really waiting for a good home, and if parso 
> or LibCST were to decide to incorporate a fork of it and develop it into a 
> high quality parser generator for Python-like languages that would be great. 
> I wouldn't worry much about the duplication of code -- the Python generator 
> in the CPython Tools directory is only used for one purpose, and that is to 
> produce the meta-parser (the parser for grammars) from the meta-grammar. And 
> I would happily stop developing the original pegen once a fork is being 
> developed.

Thanks, this is all very clarifying! I hadn't even found the original
gvanrossum/pegen repo, and was just looking at the CPython PR for PEP
617. Clearly I haven't been following this work closely.

> Another option would be to just improve the python generator in the original 
> pegen repo to satisfy the needs of parso and LibCST. Reading the blurb for 
> parso it looks like it really just parses *Python*, which is less ambitious 
> than pegen. But it also seems to support error recovery, which currently 
> isn't part of pegen. (However, we've thought about it.) Anyway, regardless of 
> how exactly this is structured someone will probably have to take over 
> development and support. Pegen started out as a hobby project to educate 
> myself about PEG parsers. Then I wrote a bunch of blog posts about my 
> approach, and finally I started working on using it to generate a replacement 
> for the old pgen-based parser. But I never found the time to make it an 
> appealing parser generator tool for other languages, even though that was on 
> my mind as a future possibility. It will take some time t

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Guido van Rossum]

Great! Please submit a PR to update the [lib]2to3 docs and CC me
(@gvanrossum).

While perhaps it wouldn't hurt if the PEP mentioned lib2to3, it was just
accepted by the Steering Council without such language, and I wouldn't want
to imply that the SC agrees with everything I said. So I still think we
ought to deal with lib2to3 independently (and no, it won't need its own PEP
:-). A reasonable option would be to just deprecate it and recommend people
use parso, LibCST or something else (I wouldn't recommend pegen in its
current form yet).

On Tue, Apr 21, 2020 at 6:21 PM Carl Meyer  wrote:

> On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum 
> wrote:
> >
> > Note that, while there is indeed a docs page about 2to3, the only docs
> for lib2to3 in the standard library reference are a link to the source code
> and a single "Note: The lib2to3 API should be considered unstable and may
> change drastically in the future."
> >
> > Fortunately,  in order to support the 2to3 application, lib2to3 doesn't
> need to change, because the syntax of Python 2 is no longer changing. :-)
> Choosing to remove 2to3 is an independent decision. And lib2to3 does not
> depend in any way on the old parser module. (It doesn't even use the
> standard tokenize module, but incorporates its own version that is slightly
> tweaked to support Python 2.)
>
> Indeed! Thanks for clarifying, I now recall that I already knew it
> doesn't, but forgot.
>
> The docs page for 2to3 does currently say "lib2to3 could also be
> adapted to custom applications in which Python code needs to be edited
> automatically." Perhaps at least this sentence should be removed, and
> maybe also replaced with a clearer note that lib2to3 not only has an
> unstable API, but also should not necessarily be expected to continue
> to parse future Python versions, and thus building tools on top of it
> should be discouraged rather than recommended. (Maybe even use the
> word "deprecated.") Happy to submit a PR for this if you agree it's
> warranted.
>
> It still seems to me that it wouldn't hurt for PEP 617 itself to also
> mention this shift in lib2to3's effective status (from "available but
> no API stability guarantee" to "probably will not parse future Python
> versions") as one of its indirect effects.
>
> > You've mentioned a few different tools that already use different
> technologies: LibCST depends on parso which has a fork of pgen2, lib2to3
> which has the original pgen2. I wonder if this would be an opportunity to
> move such parsing support out of the standard library completely. There are
> already two versions of pegen, but neither is in the standard library:
> there is the original pegen repo which is where things started, and there
> is a fork of that code in the CPython Tools directory (not yet in the
> upstream repo, but see PR 19503).
> >
> > The pegen tool has two generators, one generating C code and one
> generating Python code. I think that the C generator is really only
> relevant for CPython itself: it relies on the builtin tokenizer (the one
> written in C, not the stdlib tokenize.py) and the generated C code depends
> on many internal APIs. In fact the C generator in the original pegen repo
> doesn't work with Python 3.9 because those internal APIs are no longer
> exported. (It also doesn't work with Python 3.7 or older because it makes
> critical use of the walrus operator. :-) Also, once we started getting
> serious about replacing the old parser, we worked exclusively on the C
> generator in the CPython Tools directory, so the version in the original
> pegen repo is lagging quite a bit behind (is is the Python grammar in that
> repo). But as I said you're not gonna need it.
> >
> > On the other hand, the Python generator is designed to be flexible, and
> while it defaults to using the stdlib tokenize.py tokenizer, you can easily
> hook up your own. Putting this version in the stdlib would be a mistake,
> because the code is pretty immature; it is really waiting for a good home,
> and if parso or LibCST were to decide to incorporate a fork of it and
> develop it into a high quality parser generator for Python-like languages
> that would be great. I wouldn't worry much about the duplication of code --
> the Python generator in the CPython Tools directory is only used for one
> purpose, and that is to produce the meta-parser (the parser for grammars)
> from the meta-grammar. And I would happily stop developing the original
> pegen once a fork is being developed.
>
> Thanks, this is all very clarifying! I hadn't even found the original
> gvanrossum/pegen repo, and was just looking at the CPython PR for PEP
> 617. Clearly I haven't been following this work closely.
>
> > Another option would be to just improve the python generator in the
> original pegen repo to satisfy the needs of parso and LibCST. Reading the
> blurb for parso it looks like it really just parses *Python*, which is less
> ambitious than pegen. But it also seems to support error re

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Gregory P. Smith]

Could we go ahead and mark lib2to3 as Pending Deprecation in 3.9 so we can
get it out of the stdlib by 3.11 or 3.12?

lib2to3 is the basis of all sorts of general source code manipulation
tooling.  Its name and original reason d'etre have moved on.  It is
actively used to parse and rewrite Python 3 code all the time.  yapf uses
it, black uses a fork of it.  Other Python code manipulation tooling uses
it.  Modernize like fixers are useful for all sorts of cleanups.

IMNSHO it would be better if lib2to3 were *not* in the stdlib anymore -
Black already chose to fork lib2to3
.  So given that it is
eventually not going to be able to parse future syntax, the better answer
seems like deprecation, putting the final version up on PyPI and letting
any descendants of it live on PyPI where they can get more active care than
a stdlib module ever does.

-gps


On Tue, Apr 21, 2020 at 6:58 PM Guido van Rossum  wrote:

> Great! Please submit a PR to update the [lib]2to3 docs and CC me
> (@gvanrossum).
>
> While perhaps it wouldn't hurt if the PEP mentioned lib2to3, it was just
> accepted by the Steering Council without such language, and I wouldn't want
> to imply that the SC agrees with everything I said. So I still think we
> ought to deal with lib2to3 independently (and no, it won't need its own PEP
> :-). A reasonable option would be to just deprecate it and recommend people
> use parso, LibCST or something else (I wouldn't recommend pegen in its
> current form yet).
>
> On Tue, Apr 21, 2020 at 6:21 PM Carl Meyer  wrote:
>
>> On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum 
>> wrote:
>> >
>> > Note that, while there is indeed a docs page about 2to3, the only docs
>> for lib2to3 in the standard library reference are a link to the source code
>> and a single "Note: The lib2to3 API should be considered unstable and may
>> change drastically in the future."
>> >
>> > Fortunately,  in order to support the 2to3 application, lib2to3 doesn't
>> need to change, because the syntax of Python 2 is no longer changing. :-)
>> Choosing to remove 2to3 is an independent decision. And lib2to3 does not
>> depend in any way on the old parser module. (It doesn't even use the
>> standard tokenize module, but incorporates its own version that is slightly
>> tweaked to support Python 2.)
>>
>> Indeed! Thanks for clarifying, I now recall that I already knew it
>> doesn't, but forgot.
>>
>> The docs page for 2to3 does currently say "lib2to3 could also be
>> adapted to custom applications in which Python code needs to be edited
>> automatically." Perhaps at least this sentence should be removed, and
>> maybe also replaced with a clearer note that lib2to3 not only has an
>> unstable API, but also should not necessarily be expected to continue
>> to parse future Python versions, and thus building tools on top of it
>> should be discouraged rather than recommended. (Maybe even use the
>> word "deprecated.") Happy to submit a PR for this if you agree it's
>> warranted.
>>
>> It still seems to me that it wouldn't hurt for PEP 617 itself to also
>> mention this shift in lib2to3's effective status (from "available but
>> no API stability guarantee" to "probably will not parse future Python
>> versions") as one of its indirect effects.
>>
>> > You've mentioned a few different tools that already use different
>> technologies: LibCST depends on parso which has a fork of pgen2, lib2to3
>> which has the original pgen2. I wonder if this would be an opportunity to
>> move such parsing support out of the standard library completely. There are
>> already two versions of pegen, but neither is in the standard library:
>> there is the original pegen repo which is where things started, and there
>> is a fork of that code in the CPython Tools directory (not yet in the
>> upstream repo, but see PR 19503).
>> >
>> > The pegen tool has two generators, one generating C code and one
>> generating Python code. I think that the C generator is really only
>> relevant for CPython itself: it relies on the builtin tokenizer (the one
>> written in C, not the stdlib tokenize.py) and the generated C code depends
>> on many internal APIs. In fact the C generator in the original pegen repo
>> doesn't work with Python 3.9 because those internal APIs are no longer
>> exported. (It also doesn't work with Python 3.7 or older because it makes
>> critical use of the walrus operator. :-) Also, once we started getting
>> serious about replacing the old parser, we worked exclusively on the C
>> generator in the CPython Tools directory, so the version in the original
>> pegen repo is lagging quite a bit behind (is is the Python grammar in that
>> repo). But as I said you're not gonna need it.
>> >
>> > On the other hand, the Python generator is designed to be flexible, and
>> while it defaults to using the stdlib tokenize.py tokenizer, you can easily
>> hook up your own. Putting this version in the stdlib would be a mistake,
>> because th

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Gregory P. Smith]

On Tue, Apr 21, 2020 at 9:35 PM Gregory P. Smith  wrote:

> Could we go ahead and mark lib2to3 as Pending Deprecation in 3.9 so we can
> get it out of the stdlib by 3.11 or 3.12?
>

I'm going ahead and tracking the idea in https://bugs.python.org/issue40360.


>
> lib2to3 is the basis of all sorts of general source code manipulation
> tooling.  Its name and original reason d'etre have moved on.  It is
> actively used to parse and rewrite Python 3 code all the time.  yapf uses
> it, black uses a fork of it.  Other Python code manipulation tooling uses
> it.  Modernize like fixers are useful for all sorts of cleanups.
>
> IMNSHO it would be better if lib2to3 were *not* in the stdlib anymore -
> Black already chose to fork lib2to3
> .  So given that it is
> eventually not going to be able to parse future syntax, the better answer
> seems like deprecation, putting the final version up on PyPI and letting
> any descendants of it live on PyPI where they can get more active care than
> a stdlib module ever does.
>
> -gps
>
>
> On Tue, Apr 21, 2020 at 6:58 PM Guido van Rossum  wrote:
>
>> Great! Please submit a PR to update the [lib]2to3 docs and CC me
>> (@gvanrossum).
>>
>> While perhaps it wouldn't hurt if the PEP mentioned lib2to3, it was just
>> accepted by the Steering Council without such language, and I wouldn't want
>> to imply that the SC agrees with everything I said. So I still think we
>> ought to deal with lib2to3 independently (and no, it won't need its own PEP
>> :-). A reasonable option would be to just deprecate it and recommend people
>> use parso, LibCST or something else (I wouldn't recommend pegen in its
>> current form yet).
>>
>> On Tue, Apr 21, 2020 at 6:21 PM Carl Meyer  wrote:
>>
>>> On Sat, Apr 18, 2020 at 10:38 PM Guido van Rossum 
>>> wrote:
>>> >
>>> > Note that, while there is indeed a docs page about 2to3, the only docs
>>> for lib2to3 in the standard library reference are a link to the source code
>>> and a single "Note: The lib2to3 API should be considered unstable and may
>>> change drastically in the future."
>>> >
>>> > Fortunately,  in order to support the 2to3 application, lib2to3
>>> doesn't need to change, because the syntax of Python 2 is no longer
>>> changing. :-) Choosing to remove 2to3 is an independent decision. And
>>> lib2to3 does not depend in any way on the old parser module. (It doesn't
>>> even use the standard tokenize module, but incorporates its own version
>>> that is slightly tweaked to support Python 2.)
>>>
>>> Indeed! Thanks for clarifying, I now recall that I already knew it
>>> doesn't, but forgot.
>>>
>>> The docs page for 2to3 does currently say "lib2to3 could also be
>>> adapted to custom applications in which Python code needs to be edited
>>> automatically." Perhaps at least this sentence should be removed, and
>>> maybe also replaced with a clearer note that lib2to3 not only has an
>>> unstable API, but also should not necessarily be expected to continue
>>> to parse future Python versions, and thus building tools on top of it
>>> should be discouraged rather than recommended. (Maybe even use the
>>> word "deprecated.") Happy to submit a PR for this if you agree it's
>>> warranted.
>>>
>>> It still seems to me that it wouldn't hurt for PEP 617 itself to also
>>> mention this shift in lib2to3's effective status (from "available but
>>> no API stability guarantee" to "probably will not parse future Python
>>> versions") as one of its indirect effects.
>>>
>>> > You've mentioned a few different tools that already use different
>>> technologies: LibCST depends on parso which has a fork of pgen2, lib2to3
>>> which has the original pgen2. I wonder if this would be an opportunity to
>>> move such parsing support out of the standard library completely. There are
>>> already two versions of pegen, but neither is in the standard library:
>>> there is the original pegen repo which is where things started, and there
>>> is a fork of that code in the CPython Tools directory (not yet in the
>>> upstream repo, but see PR 19503).
>>> >
>>> > The pegen tool has two generators, one generating C code and one
>>> generating Python code. I think that the C generator is really only
>>> relevant for CPython itself: it relies on the builtin tokenizer (the one
>>> written in C, not the stdlib tokenize.py) and the generated C code depends
>>> on many internal APIs. In fact the C generator in the original pegen repo
>>> doesn't work with Python 3.9 because those internal APIs are no longer
>>> exported. (It also doesn't work with Python 3.7 or older because it makes
>>> critical use of the walrus operator. :-) Also, once we started getting
>>> serious about replacing the old parser, we worked exclusively on the C
>>> generator in the CPython Tools directory, so the version in the original
>>> pegen repo is lagging quite a bit behind (is is the Python grammar in that
>>> repo). But as I said you're not gonna need it.
>>> >

[Python-Dev] Re: PEP 554 comments

[Greg Ewing]

On 22/04/20 3:57 am, Eric Snow wrote:

The main difference is that the PEP also provides an way to explicitly
release or close a channel.  Providing just "close()" would mean one
interpreter could stomp on all other interpreters' use of a channel.


What I'm suggesting is that close() should do what the
PEP defines release() as doing, and release() shouldn't
exist.

I don't see why an interpreter needs the ability to close
a channel for any *other* interpreter. There is no such
ability for files and pipes.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZVMQFHRJWSE2PMVICB5ZFLY6NVZBQ3ZN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Greg Ewing]

On 22/04/20 4:21 am, Sebastian Berg wrote:

Sure, but this is very different. You can still use NumPy in a project
using asyncio. You are _not_ able to use NumPy in a project using
subinterpreters.


To put it another way, the moment you start using subinterpreters,
the set of extension modules you are able to use will shrink
*enormously*.

And if I understand correctly, you won't get any nice "This
module does not support subinterpreters" exception if you
import an incompatible module -- just an obscure crash,
probably of the core-dumping variety.

To me this would feel like programming in the middle of a
lethal minefield.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OLTRZWS5SRBXXLB25C44DE5MTTFZDWJS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Glenn Linderman]

On 4/21/2020 10:26 PM, Greg Ewing wrote:

And if I understand correctly, you won't get any nice "This
module does not support subinterpreters" exception if you
import an incompatible module -- just an obscure crash,
probably of the core-dumping variety. 


This sounds fixable: modules that support subinterpreters should set a 
flag saying so, and the either the load of a non-flagged module when 
subinterpreters are in use, or the initiation of a subinterpreter when a 
non-flagged module has been loaded, should raise.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H3TCU43YUSUMYUMWBA6PFZZJ7AG5FZCD/
Code of Conduct: http://python.org/psf/codeofconduct/