Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On Dec 6, 2017 21:14, "Guido van Rossum" wrote: OK, then please just change the PEP's Version: header to 3.8. Will do. Have a nice vacation! :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
OK, then please just change the PEP's Version: header to 3.8. On Wed, Dec 6, 2017 at 7:57 PM, Eric Snow wrote: > > > On Dec 6, 2017 20:31, "Guido van Rossum" wrote: > > If the point is just to be able to test the existing API better, no PEP is > needed, right? It would be an unsupported, undocumented API. > > > In the short term that's one major goal. In the long term the > functionality provided by the PEP is a prerequisite for other > concurrency-related features, and targeting 3.8 for that is fine. :) > > -eric > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On Dec 6, 2017 20:31, "Guido van Rossum" wrote: If the point is just to be able to test the existing API better, no PEP is needed, right? It would be an unsupported, undocumented API. In the short term that's one major goal. In the long term the functionality provided by the PEP is a prerequisite for other concurrency-related features, and targeting 3.8 for that is fine. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
If the point is just to be able to test the existing API better, no PEP is needed, right? It would be an unsupported, undocumented API. On Wed, Dec 6, 2017 at 7:22 PM, Nick Coghlan wrote: > On 7 December 2017 at 12:46, Guido van Rossum wrote: > > So you're okay with putting this off till (at least) 3.8? That sounds > good > > to me, given that I'd like to go on vacation soon. > > Eric reminded me off-list that we'd like to at least add the lower > level _interpreters API for the benefit of the test suite - right now, > all of our subinterpreter testing needs to be run through either > test_embed or test_capi, which is annoying enough that we end up > simply not testing the subinterpreter functionality properly (in > practice, we're relying heavily on the regression test suites for > mod_wsgi and JEP to find any problems we inadvertently introduce when > refactoring CPython's internals). > > If we were to put that under test.support._interpreters for 3.7, we'd > be able to make it clear that we're in "Even more experimental than > provisional API status would account for" territory, while still > enabling the improved testing and accessibility for experimentation > that we're after in order to make some better informed API design > proposals for Python 3.8. > > Regards, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On 7 December 2017 at 12:46, Guido van Rossum wrote: > So you're okay with putting this off till (at least) 3.8? That sounds good > to me, given that I'd like to go on vacation soon. Eric reminded me off-list that we'd like to at least add the lower level _interpreters API for the benefit of the test suite - right now, all of our subinterpreter testing needs to be run through either test_embed or test_capi, which is annoying enough that we end up simply not testing the subinterpreter functionality properly (in practice, we're relying heavily on the regression test suites for mod_wsgi and JEP to find any problems we inadvertently introduce when refactoring CPython's internals). If we were to put that under test.support._interpreters for 3.7, we'd be able to make it clear that we're in "Even more experimental than provisional API status would account for" territory, while still enabling the improved testing and accessibility for experimentation that we're after in order to make some better informed API design proposals for Python 3.8. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
So you're okay with putting this off till (at least) 3.8? That sounds good to me, given that I'd like to go on vacation soon. On Wed, Dec 6, 2017 at 5:04 PM, Nick Coghlan wrote: > On 7 December 2017 at 01:50, Guido van Rossum wrote: > > Sorry to burst your bubble, but I have not followed any of the discussion > > and I am actually very worried about this topic. I don't think I will be > > able to make time for this before the 3.7b1 feature freeze. > > I think that will be OK, as it will encourage us to refactor Eric's > branch into two distinct pieces in the meantime: exposing any needed C > API elements that aren't currently visible as > "nominally-private-but-linkable-if-you're-prepared-to-cope-with-potential- > instability" > interfaces, and then a pip-installable extension module that adds the > Python level API. > > We won't be able to experiment with ideas like removing GIL sharing > between subinterpreters that way, but we'll be able to work on the > semantics of the user facing API design, and enable experimentation > with things like CSP and Actor-based programming backed by stronger > memory separation than is offered by Python threads. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On 7 December 2017 at 01:50, Guido van Rossum wrote: > Sorry to burst your bubble, but I have not followed any of the discussion > and I am actually very worried about this topic. I don't think I will be > able to make time for this before the 3.7b1 feature freeze. I think that will be OK, as it will encourage us to refactor Eric's branch into two distinct pieces in the meantime: exposing any needed C API elements that aren't currently visible as "nominally-private-but-linkable-if-you're-prepared-to-cope-with-potential-instability" interfaces, and then a pip-installable extension module that adds the Python level API. We won't be able to experiment with ideas like removing GIL sharing between subinterpreters that way, but we'll be able to work on the semantics of the user facing API design, and enable experimentation with things like CSP and Actor-based programming backed by stronger memory separation than is offered by Python threads. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
Sorry to burst your bubble, but I have not followed any of the discussion and I am actually very worried about this topic. I don't think I will be able to make time for this before the 3.7b1 feature freeze. On Tue, Dec 5, 2017 at 6:51 PM, Eric Snow wrote: > Hi all, > > I've finally updated PEP 554. Feedback would be most welcome. The > PEP is in a pretty good place now and I hope to we're close to a > decision to accept it. :) > > In addition to resolving the open questions, I've also made the > following changes to the PEP: > > * put an API summary at the top and moved the full API description down > * add the "is_shareable()" function to indicate if an object can be shared > * added None as a shareable object > > Regarding the open questions: > > * "Leaking exceptions across interpreters" > > I chose to go with an approach that effectively creates a > traceback.TracebackException proxy of the original exception, wraps > that in a RuntimeError, and raises that in the calling interpreter. > Raising an exception that safely preserves the original exception and > traceback seems like the most intuitive behavior (to me, as a user). > The only alternative that made sense is to fully duplicate the > exception and traceback (minus stack frames) in the calling > interpreter, which is probably overkill and likely to be confusing. > > * "Initial support for buffers in channels" > > I chose to add a "SendChannel.send_buffer(obj)" method for this. > Supporting buffer objects from the beginning makes sense, opening good > experimentation opportunities for a valuable set of users. Supporting > buffer objects separately and explicitly helps set clear expectations > for users. I decided not to go with a separate class (e.g. > MemChannel) as it didn't seem like there's enough difference to > warrant keeping them strictly separate. > > FWIW, I'm still strongly in favor of support for passing (copies of) > bytes objects via channels. Passing objects to SendChannel.send() is > obvious. Limiting it, for now, to bytes (and None) helps us avoid > tying ourselves strongly to any particular implementation (it seems > like all the reservations were relative to the implementation). So I > do not see a reason to wait. > > * "Pass channels explicitly to run()?" > > I've applied the suggested solution (make "channels" an explicit > keyword argument). > > -eric > > > I've include the latest full text > (https://www.python.org/dev/peps/pep-0554/) below: > > + > > PEP: 554 > Title: Multiple Interpreters in the Stdlib > Author: Eric Snow > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 2017-09-05 > Python-Version: 3.7 > Post-History: 07-Sep-2017, 08-Sep-2017, 13-Sep-2017, 05-Dec-2017 > > > Abstract > > > CPython has supported multiple interpreters in the same process (AKA > "subinterpreters") since version 1.5. The feature has been available > via the C-API. [c-api]_ Subinterpreters operate in > `relative isolation from one another `_, which > provides the basis for an > `alternative concurrency model `_. > > This proposal introduces the stdlib ``interpreters`` module. The module > will be `provisional `_. It exposes the basic > functionality of subinterpreters already provided by the C-API, along > with new functionality for sharing data between interpreters. > > > Proposal > > > The ``interpreters`` module will be added to the stdlib. It will > provide a high-level interface to subinterpreters and wrap a new > low-level ``_interpreters`` (in the same was as the ``threading`` > module). See the `Examples`_ section for concrete usage and use cases. > > Along with exposing the existing (in CPython) subinterpreter support, > the module will also provide a mechanism for sharing data between > interpreters. This mechanism centers around "channels", which are > similar to queues and pipes. > > Note that *objects* are not shared between interpreters since they are > tied to the interpreter in which they were created. Instead, the > objects' *data* is passed between interpreters. See the `Shared data`_ > section for more details about sharing between interpreters. > > At first only the following types will be supported for sharing: > > * None > * bytes > * PEP 3118 buffer objects (via ``send_buffer()``) > > Support for other basic types (e.g. int, Ellipsis) will be added later. > > API summary for interpreters module > --- > > Here is a summary of the API for the ``interpreters`` module. For a > more in-depth explanation of the proposed classes and functions, see > the `"interpreters" Module API`_ section below. > > For creating and using interpreters: > > +--+ > --+ > | signature| description > | > ++=+ > ==+ > | list_all() -> [Inte
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On Dec 5, 2017 23:49, "Nick Coghlan" wrote: Nice updates! I like this version. Great! :) My one suggestion here would be to consider a dedicated exception type like "interpreters.SubinterpreterError", rather than re-using RuntimeError directly. That way you can put the extracted traceback on a named attribute, and retain the option of potentially adding subinterpreter awareness to the traceback module in the future. Yeah, I already have a deferred idea item for this. :). TBH, I was on the fence about a dedicated exception type, so you've nudged me on board. :) -eric ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On 6 December 2017 at 12:51, Eric Snow wrote: > Hi all, > > I've finally updated PEP 554. Feedback would be most welcome. The > PEP is in a pretty good place now and I hope to we're close to a > decision to accept it. :) Nice updates! I like this version. > In addition to resolving the open questions, I've also made the > following changes to the PEP: > > * put an API summary at the top and moved the full API description down > * add the "is_shareable()" function to indicate if an object can be shared > * added None as a shareable object > > Regarding the open questions: > > * "Leaking exceptions across interpreters" > > I chose to go with an approach that effectively creates a > traceback.TracebackException proxy of the original exception, wraps > that in a RuntimeError, and raises that in the calling interpreter. > Raising an exception that safely preserves the original exception and > traceback seems like the most intuitive behavior (to me, as a user). > The only alternative that made sense is to fully duplicate the > exception and traceback (minus stack frames) in the calling > interpreter, which is probably overkill and likely to be confusing. My one suggestion here would be to consider a dedicated exception type like "interpreters.SubinterpreterError", rather than re-using RuntimeError directly. That way you can put the extracted traceback on a named attribute, and retain the option of potentially adding subinterpreter awareness to the traceback module in the future. > * "Initial support for buffers in channels" > > I chose to add a "SendChannel.send_buffer(obj)" method for this. > Supporting buffer objects from the beginning makes sense, opening good > experimentation opportunities for a valuable set of users. Supporting > buffer objects separately and explicitly helps set clear expectations > for users. I decided not to go with a separate class (e.g. > MemChannel) as it didn't seem like there's enough difference to > warrant keeping them strictly separate. > > FWIW, I'm still strongly in favor of support for passing (copies of) > bytes objects via channels. Passing objects to SendChannel.send() is > obvious. Limiting it, for now, to bytes (and None) helps us avoid > tying ourselves strongly to any particular implementation (it seems > like all the reservations were relative to the implementation). So I > do not see a reason to wait. Aye, the split sending API with a merged receive API works for me. > * "Pass channels explicitly to run()?" > > I've applied the suggested solution (make "channels" an explicit > keyword argument). Cool. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v4 (new interpreters module)
On 12/06/2017 03:51 AM, Eric Snow wrote: Hi all, I've finally updated PEP 554. Feedback would be most welcome. The PEP is in a pretty good place now and I hope to we're close to a decision to accept it. :) [...] C-extension opt-in/opt-out -- By using the ``PyModuleDef_Slot`` introduced by PEP 489, we could easily add a mechanism by which C-extension modules could opt out of support for subinterpreters. Then the import machinery, when operating in a subinterpreter, would need to check the module for support. It would raise an ImportError if unsupported. > Alternately we could support opting in to subinterpreter support. However, that would probably exclude many more modules (unnecessarily) than the opt-out approach. Currently it's already opt-in, as modules that use PyModuleDef are expected to support subinterpreters: https://www.python.org/dev/peps/pep-0489/#subinterpreters-and-interpreter-reloading [...] .. [global-atexit] https://bugs.python.org/issue6531 Oh dear; there's now also https://bugs.python.org/issue31901 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
