Re: [Python-Dev] PEP 557: Data Classes

2017-09-13 Thread Eric V. Smith 

On 9/10/2017 11:08 PM, Nathaniel Smith wrote:

Hi Eric,

A few quick comments:

Why do you even have a hash= argument on individual fields? For the 
whole class, I can imagine you might want to explicitly mark a whole 
class as unhashable, but it seems like the only thing you can do with 
the field-level hash= argument is to create a class where the __hash__ 
and __eq__ take different fields into account, and why would you ever 
want that?


See the discussion at 
https://github.com/ericvsmith/dataclasses/issues/44 for why we're 
keeping the field-level hash function.


Quick version, from Guido: "There's a legitimate reason for having a 
field in the eq but not in the hash. After all hash is always followed 
by an eq check and it is totally legit for the eq check to say two 
objects are not equal even though their hashes are equal."


This would be in the case where a field should be used for equality 
testing, but computing its hash is expensive.


Eric.
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

2017-09-13 Thread Larry Hastings 


On 09/12/2017 12:38 AM, Larry Hastings wrote:

On 09/11/2017 07:22 PM, Guido van Rossum wrote:


The prototype is linked to from the PEP; for your convenience
here's a link:

https://github.com/larryhastings/cpython/tree/module-properties


I found that link in the PEP, but it's just a branch of a fork of 
cpython. It would be easier to review the prototype as a PR to 
upstream cpython.


Okay, I'll make a PR tomorrow and post a reply here (and update the PEP).


PR is here:

   https://github.com/python/cpython/pull/3534


Cheers,


//arry/
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Reminder: snapshots and releases coming up in the next several days

2017-09-13 Thread Ned Deily 
Lots of releases coming up soon!

2017-09-16:
- Python 2.7.14 final release

2017-09-18 1200 UTC cutoff:
- Python 3.6.3 rc1
- Python 3.3.7 final release (following by retirement)

If you know of any issues that you feel need to be addressed in these releases, 
please make sure there is an open issue on the bug tracker set to "release 
blocker".

Also on 2017-09-18:
- Python 3.7.0 alpha 1

This will be the first preview snapshot of 3.7.0.  It's still very early in the 
development cycle for 3.7.  The main purpose of early alpha releases is to 
exercise the build and release process and to make it easier for Windows and 
Macs users to help test.  Many new features and build changes are yet to appear 
in subsequent releases prior to feature code cutoff on 2018-01-29 at 3.7.0b1.

Thanks for your help!

--Ned

--
  Ned Deily
  [email protected] -- []

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: snapshots and releases coming up in the next several days

2017-09-13 Thread Fred Drake 
On Wed, Sep 13, 2017 at 12:35 PM, Ned Deily  wrote:
> Lots of releases coming up soon!

There's a "Python Release Schedule" calendar on Google Calendar that
used to be maintained, but that appears to have been dropped, though I
found it useful.

Is there any sort of calendar feed available with this schedule that's
currently maintained?


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reminder: snapshots and releases coming up in the next several days

2017-09-13 Thread Ned Deily 
I know the issue of the Google Calendar feed has come up before and we should 
either maintain it or remove it from the web site. I don't know who has or had 
the "keys" to it. I'm traveling the next couple of days but I'll look into it 
afterwards. If anyone has info about it, please contact me directly.  AFAIK, 
there is no other release calendar feed currently.

  --
Ned Deily
[email protected] -- []



> On Sep 13, 2017, at 09:57, Fred Drake  wrote:
> 
>> On Wed, Sep 13, 2017 at 12:35 PM, Ned Deily  wrote:
>> Lots of releases coming up soon!
> 
> There's a "Python Release Schedule" calendar on Google Calendar that
> used to be maintained, but that appears to have been dropped, though I
> found it useful.
> 
> Is there any sort of calendar feed available with this schedule that's
> currently maintained?
> 
> 
>  -Fred
> 
> -- 
> Fred L. Drake, Jr.
> "A storm broke loose in my mind."  --Albert Einstein

___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] parallelizing

2017-09-13 Thread Chris Barker 
This really isn't the place to ask this kind of question.

If you want to know how to do something with python, try python-users ,
stack overflow, etc.

If you have an idea about a new feature you think python could have, then
the python-ideas list is the place for that. But if you want anyone to take
it seriously, it should be a better formed idea before you post there.

But:

On Tue, Sep 12, 2017 at 4:43 PM, Matthieu Bec  wrote:

> There are times when you deal with completely independent input/output
> 'pipes' - where parallelizing would really help speed things up.
>
> Can't there be a way to capture that idiom and multi thread it in the
> language itself?
>
> Example:
>
> loop:
>
> read an XML
>
> produce a JSON like
>

Regular old threading works fine for this:

import time
import random
import threading


def process(infile, outfile):
"fake function to simulate a process that takes a random amount of time"
time.sleep(random.random())
print("processing: {} to make {}".format(infile, outfile))


for i in range(10):
threading.Thread(target=process, args=("file%i.xml" % i, "file%i.xml" %
i)).start()


It gets complicated if you need to pass information back and forth, or
worry about race conditions, or manage a queue, or 

But just running a nice self-contained thread safe function in another
thread is pretty straightforward.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

[email protected]
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

2017-09-13 Thread Sven R. Kunze 

Why not adding both? Properties do have their uses as does __getattr__.

Cheers,
Sven

On 13.09.2017 11:43, Larry Hastings wrote:


On 09/12/2017 12:38 AM, Larry Hastings wrote:

On 09/11/2017 07:22 PM, Guido van Rossum wrote:


The prototype is linked to from the PEP; for your convenience
here's a link:

https://github.com/larryhastings/cpython/tree/module-properties


I found that link in the PEP, but it's just a branch of a fork of 
cpython. It would be easier to review the prototype as a PR to 
upstream cpython.


Okay, I'll make a PR tomorrow and post a reply here (and update the PEP).


PR is here:

https://github.com/python/cpython/pull/3534


Cheers,


//arry/


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

2017-09-13 Thread Guido van Rossum 
 > Why not adding both? Properties do have their uses as does __getattr__.

In that case I would just add __getattr__ to module.c, and add a recipe or
perhaps a utility module that implements a __getattr__ you can put into
your module if you want @property support. That way you can have both but
you only need a little bit of code in module.c to check for __getattr__ and
call it when you'd otherwise raise AttributeError.

-- 
--Guido van Rossum (python.org/~guido )
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] A reminder for PEP owners

2017-09-13 Thread Guido van Rossum 
I know there's a lot of excitement around lots of new ideas. And the 3.7
feature freeze is looming (January is in a few months). But someone has to
review and accept all those PEPs, and I can't do it all by myself.

If you want your proposal to be taken seriously, you need to include a
summary of the discussion on the mailing list (including objections, even
if you disagree!) in your PEP, e.g. as an extended design rationale or
under the Rejected Ideas heading.

If you don't do this you risk having to repeat yourself -- also you risk
having your PEP rejected, because at this point there's no way I am going
to read all the discussions.

-- 
--Guido van Rossum (python.org/~guido )
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] parallelizing

2017-09-13 Thread Matthieu Bec 

Thank you, I'll take your advice.

Regarding your example, I think it gives the illusion to work because 
sleep() is GIL aware under the hood.


I don't think it works for process() that mainly runs bytecode, because 
of the GIL.


Sorry if I wrongly thought that was a language level discussion.

Regards,

Matthieu


On 9/13/17 10:49 AM, Chris Barker wrote:

This really isn't the place to ask this kind of question.

If you want to know how to do something with python, try python-users 
, stack overflow, etc.


If you have an idea about a new feature you think python could have, 
then the python-ideas list is the place for that. But if you want 
anyone to take it seriously, it should be a better formed idea before 
you post there.


But:

On Tue, Sep 12, 2017 at 4:43 PM, Matthieu Bec > wrote:


There are times when you deal with completely independent
input/output 'pipes' - where parallelizing would really help speed
things up.

Can't there be a way to capture that idiom and multi thread it in
the language itself?

Example:

loop:

read an XML

produce a JSON like


Regular old threading works fine for this:

import time
import random
import threading


def process(infile, outfile):
"fake function to simulate a process that takes a random amount of 
time"

time.sleep(random.random())
print("processing: {} to make {}".format(infile, outfile))


for i in range(10):
threading.Thread(target=process, args=("file%i.xml" % i, 
"file%i.xml" % i)).start()



It gets complicated if you need to pass information back and forth, or 
worry about race conditions, or manage a queue, or 


But just running a nice self-contained thread safe function in another 
thread is pretty straightforward.


-CHB



--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

[email protected] 


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

2017-09-13 Thread Nathaniel Smith 
On Wed, Sep 13, 2017 at 11:49 AM, Guido van Rossum  wrote:
>  > Why not adding both? Properties do have their uses as does __getattr__.
>
> In that case I would just add __getattr__ to module.c, and add a recipe or
> perhaps a utility module that implements a __getattr__ you can put into your
> module if you want @property support. That way you can have both but you
> only need a little bit of code in module.c to check for __getattr__ and call
> it when you'd otherwise raise AttributeError.

Unfortunately I don't think this works. If there's a @property object
present in the module's instance dict, then __getattribute__ will
return it directly instead of calling __getattr__.

(I guess for full property emulation you'd also need to override
__setattr__ and __dir__, but I don't know how important that is.)

We could consider letting modules overload __getattribute__ instead of
__getattr__, but I don't think this is viable either -- a key feature
of __getattr__ is that it doesn't add overhead to normal lookups. If
you implement deprecation warnings by overloading __getattribute__,
then it makes all your users slower, even the ones who never touch the
deprecated attributes. __getattr__ is much better than
__getattribute__ for this purpose.

Alternatively we can have a recipe that implements @property support
using __class__ assignment and overriding
__getattribute__/__setattr__/__dir__, so instead of 'from
module_helper.property_emulation import __getattr__' it'd be 'from
module_helper import enable_property_emulation;
enable_property_emulation(__name__)'. Still has the slowdown problem
but it would work.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A reminder for PEP owners

2017-09-13 Thread Skip Montanaro 
> But someone has to
> review and accept all those PEPs, and I can't do it all by myself.

An alternate definition for BDFL is "Benevolent Delegator For Life." :-)

Skip
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

2017-09-13 Thread Guido van Rossum 
On Wed, Sep 13, 2017 at 2:00 PM, Nathaniel Smith  wrote:

> On Wed, Sep 13, 2017 at 11:49 AM, Guido van Rossum 
> wrote:
> >  > Why not adding both? Properties do have their uses as does
> __getattr__.
> >
> > In that case I would just add __getattr__ to module.c, and add a recipe
> or
> > perhaps a utility module that implements a __getattr__ you can put into
> your
> > module if you want @property support. That way you can have both but you
> > only need a little bit of code in module.c to check for __getattr__ and
> call
> > it when you'd otherwise raise AttributeError.
>
> Unfortunately I don't think this works. If there's a @property object
> present in the module's instance dict, then __getattribute__ will
> return it directly instead of calling __getattr__.
>

Hm, that's a good point. One would have to introduce some kind of
convention where you can write properties with a leading _:

@property
def _foo(): return 42

and then a utility __getattr__ like this:

def __getattr__(name):
g = globals()
name = '_' + name
if name in g:
return g[name]()
raise AttributeError(...)


> (I guess for full property emulation you'd also need to override
> __setattr__ and __dir__, but I don't know how important that is.)
>

At that point maybe __class__ assignment is better.


> We could consider letting modules overload __getattribute__ instead of
> __getattr__, but I don't think this is viable either -- a key feature
> of __getattr__ is that it doesn't add overhead to normal lookups. If
> you implement deprecation warnings by overloading __getattribute__,
> then it makes all your users slower, even the ones who never touch the
> deprecated attributes. __getattr__ is much better than
> __getattribute__ for this purpose.
>

Agreed.

Alternatively we can have a recipe that implements @property support
> using __class__ assignment and overriding
> __getattribute__/__setattr__/__dir__, so instead of 'from
> module_helper.property_emulation import __getattr__' it'd be 'from
> module_helper import enable_property_emulation;
> enable_property_emulation(__name__)'. Still has the slowdown problem
> but it would work.
>

The emulation could do something less drastic than __class__ assignment --
it could look for globals that are properties, move them into some other
dict (e.g. __properties__), and install a __getattr__ that looks things up
in that dict and calls them.

def __getattr__(name):
if name in __properties__:
return __properties__[name]()
raise AttributeError(...)

Still, proposals for sys.py notwithstanding, I'm worried that all of this
is a solution looking for a problem. Yes, it's a cute hack. But is it art?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

2017-09-13 Thread Ivan Levkivskyi 
@Guido
> One would have to introduce some kind of convention
> where you can write properties with a leading _

One doesn't even need the @property decorator in this case.
For example:

def __getattr__(name):
g = globals()
name = '_' + name
if name in g:
return g[name]()
raise AttributeError(...)

def _attr():
"do something"
return actual_object

One can even write a decorator that would change the name automatically
(but this will require a call at the end of module to unbind original
names).
The space for play is infinite. The question is what exactly is needed.
I would say that already basic customisation (like __getattr__) will be
enough.
IIUC, this will be actually a bit faster than setting __class__ since only
"customized"
attributes will be affected.

--
Ivan
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 549: Instance Properties (aka: module properties)

2017-09-13 Thread Guido van Rossum 
On Wed, Sep 13, 2017 at 3:01 PM, Ivan Levkivskyi 
wrote:

> @Guido
> > One would have to introduce some kind of convention
> > where you can write properties with a leading _
>
> One doesn't even need the @property decorator in this case.
> For example:
>
> def __getattr__(name):
> g = globals()
> name = '_' + name
> if name in g:
> return g[name]()
> raise AttributeError(...)
>
> def _attr():
> "do something"
> return actual_object
>
> One can even write a decorator that would change the name automatically
> (but this will require a call at the end of module to unbind original
> names).
> The space for play is infinite. The question is what exactly is needed.
> I would say that already basic customisation (like __getattr__) will be
> enough.
> IIUC, this will be actually a bit faster than setting __class__ since only
> "customized"
> attributes will be affected.
>

That last sentence is a key observation. Do we even know whether there are
(non-toy) things that you can do *in principle* with __class__ assignment
but which are too slow *in practice* to bother? And if yes, is __getattr__
fast enough? @property?

IMO we're still looking for applications.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] parallelizing

2017-09-13 Thread Chris Barker 
On Wed, Sep 13, 2017 at 12:11 PM, Matthieu Bec  wrote:

> Regarding your example, I think it gives the illusion to work because
> sleep() is GIL aware under the hood.
>
>
It'll work for anything -- it just may not buy you any performance.

I don't know off the top of my head if file I/O captures the GIL -- for
your example of file parsing.

> I don't think it works for process() that mainly runs bytecode, because of
> the GIL.
>
If you are trying to get around the GIL that that is a totally different
question.

But the easy way is to use multiprocessing instead:

import time
import random
import multiprocessing


def process(infile, outfile):
"fake function to simulate a process that takes a random amount of time"
time.sleep(random.random())
print("processing: {} to make {}".format(infile, outfile))

for i in range(10):
multiprocessing.Process(target=process, args=("file%i.xml" % i,
"file%i.xml" % i)).start()

More overhead creating the processes, but no more GIL issues.

Sorry if I wrongly thought that was a language level discussion.
>
>
This list is for discussion of the development of the cPython interpreter.
So this kind of discussion doesn't belong here unless/until it gets to the
point of actually implementing something.

If you have an idea as to how to improve Python, then python-ideas is the
place for that discussion.

But "there should be a way to run threads without the GIL" isn't a
well-enough formed idea to get far there

If you want to discuss further, let's take this offline.

> Can't there be a way to capture that idiom and multi thread it in the
language itself?

>
>> Example:
>>
>> loop:
>>
>> read an XML
>>
>> produce a JSON like
>>
> note about this -- code like this would be using all sorts of shared
modules. The code in those modules is going to be touched by all the
threads. There is no way the python interpreter can know which python
objects are used by what how --- the GIL is there for good (and complex)
reasons, not an easy task to avoid it. It's all using the same interpreter.

Also -- it's not easy to know what code may work OK with the GIL. intensive
computation is bad. But Python is a poor choice for that anyway.

And code that does a lot in C -- numpy, text processing, etc. may not hold
the GIL. And I/O

So for your example of parsing XML and writing JSON -- it may well do a lot
of work without holding the GIL.

No way to know but to profile it.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

[email protected]
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 554 v3 (new interpreters module)

2017-09-13 Thread Eric Snow 
I've updated PEP 554 in response to feedback.  (thanks all!)  There
are a few unresolved points (some of them added to the Open Questions
section), but the current PEP has changed enough that I wanted to get
it out there first.

Notably changed:

* the API relative to object passing has changed somewhat drastically
(hopefully simpler and easier to understand), replacing "FIFO" with
"channel"
* added an examples section
* added an open questions section
* added a rejected ideas section
* added more items to the deferred functionality section
* the rationale section has moved down below the examples

Please let me know what you think.  I'm especially interested in
feedback about the channels.  Thanks!

-eric




PEP: 554
Title: Multiple Interpreters in the Stdlib
Author: Eric Snow 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-05
Python-Version: 3.7
Post-History:


Abstract


CPython has supported subinterpreters, with increasing levels of
support, since version 1.5.  The feature has been available via the
C-API. [c-api]_  Subinterpreters operate in
`relative isolation from one another `_, which
provides the basis for an
`alternative concurrency model `_.

This proposal introduces the stdlib ``interpreters`` module.  The module
will be `provisional `_.  It exposes the basic
functionality of subinterpreters already provided by the C-API.


Proposal


The ``interpreters`` module will be added to the stdlib.  It will
provide a high-level interface to subinterpreters and wrap the low-level
``_interpreters`` module.  The proposed API is inspired by the
``threading`` module.  See the `Examples`_ section for concrete usage
and use cases.

API for interpreters


The module provides the following functions:

``list_all()``::

   Return a list of all existing interpreters.

``get_current()``::

   Return the currently running interpreter.

``create()``::

   Initialize a new Python interpreter and return it.  The
   interpreter will be created in the current thread and will remain
   idle until something is run in it.  The interpreter may be used
   in any thread and will run in whichever thread calls
   ``interp.run()``.


The module also provides the following class:

``Interpreter(id)``::

   id:

  The interpreter's ID (read-only).

   is_running():

  Return whether or not the interpreter is currently executing code.
  Calling this on the current interpreter will always return True.

   destroy():

  Finalize and destroy the interpreter.

  This may not be called on an already running interpreter.  Doing
  so results in a RuntimeError.

   run(source_str, /, **shared):

  Run the provided Python source code in the interpreter.  Any
  keyword arguments are added to the interpreter's execution
  namespace.  If any of the values are not supported for sharing
  between interpreters then RuntimeError gets raised.  Currently
  only channels (see "create_channel()" below) are supported.

  This may not be called on an already running interpreter.  Doing
  so results in a RuntimeError.

  A "run()" call is quite similar to any other function call.  Once
  it completes, the code that called "run()" continues executing
  (in the original interpreter).  Likewise, if there is any uncaught
  exception, it propagates into the code where "run()" was called.

  The big difference is that "run()" executes the code in an
  entirely different interpreter, with entirely separate state.
  The state of the current interpreter in the current OS thread
  is swapped out with the state of the target interpreter (the one
  that will execute the code).  When the target finishes executing,
  the original interpreter gets swapped back in and its execution
  resumes.

  So calling "run()" will effectively cause the current Python
  thread to pause.  Sometimes you won't want that pause, in which
  case you should make the "run()" call in another thread.  To do
  so, add a function that calls "run()" and then run that function
  in a normal "threading.Thread".

  Note that interpreter's state is never reset, neither before
  "run()" executes the code nor after.  Thus the interpreter
  state is preserved between calls to "run()".  This includes
  "sys.modules", the "builtins" module, and the internal state
  of C extension modules.

  Also note that "run()" executes in the namespace of the "__main__"
  module, just like scripts, the REPL, "-m", and "-c".  Just as
  the interpreter's state is not ever reset, the "__main__" module
  is never reset.  You can imagine concatenating the code from each
  "run()" call into one long script.  This is the same as how the
  REPL operates.

  Supported code: source text.

API for sharing data


The mechanism for pa

[Python-Dev] PEP 553, v3

2017-09-13 Thread Barry Warsaw 
Here’s an update to PEP 553, which makes $PYTHONBREAKPOINT a first class 
feature.  I’ve also updated PR #3355 with the implementation to match.

Cheers,
-Barry

PEP: 553
Title: Built-in breakpoint()
Author: Barry Warsaw 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-05
Python-Version: 3.7
Post-History: 2017-09-05, 2017-09-07, 2017-09-13


Abstract


This PEP proposes adding a new built-in function called ``breakpoint()`` which
enters a Python debugger at the point of the call.  Additionally, two new
names are added to the ``sys`` module to make the debugger pluggable.


Rationale
=

Python has long had a great debugger in its standard library called ``pdb``.
Setting a break point is commonly written like this::

foo()
import pdb; pdb.set_trace()
bar()

Thus after executing ``foo()`` and before executing ``bar()``, Python will
enter the debugger.  However this idiom has several disadvantages.

* It's a lot to type (27 characters).

* It's easy to typo.  The PEP author often mistypes this line, e.g. omitting
  the semicolon, or typing a dot instead of an underscore.

* It ties debugging directly to the choice of pdb.  There might be other
  debugging options, say if you're using an IDE or some other development
  environment.

* Python linters (e.g. flake8 [linters]_) complain about this line because it
  contains two statements.  Breaking the idiom up into two lines further
  complicates the use of the debugger,

These problems can be solved by modeling a solution based on prior art in
other languages, and utilizing a convention that already exists in Python.


Proposal


The JavaScript language provides a ``debugger`` statement [java]_ which enters
the debugger at the point where the statement appears.

This PEP proposes a new built-in function called ``breakpoint()``
which enters a Python debugger at the call site.  Thus the example
above would be written like so::

foo()
breakpoint()
bar()

Further, this PEP proposes two new name bindings for the ``sys``
module, called ``sys.breakpointhook()`` and
``sys.__breakpointhook__``.  By default, ``sys.breakpointhook()``
implements the actual importing and entry into ``pdb.set_trace()``,
and it can be set to a different function to change the debugger that
``breakpoint()`` enters.  ``sys.__breakpointhook__`` then stashes the
default value of ``sys.breakpointhook()`` to make it easy to reset.
This exactly models the existing ``sys.displayhook()`` /
``sys.__displayhook__`` and ``sys.excepthook()`` /
``sys.__excepthook__`` [hooks]_.

The signature of the built-in is ``breakpoint(*args, **kws)``.  The positional
and keyword arguments are passed straight through to ``sys.breakpointhook()``
and the signatures must match or a ``TypeError`` will be raised.  The return
from ``sys.breakpointhook()`` is passed back up to, and returned from
``breakpoint()``.  Since ``sys.breakpointhook()`` by default calls
``pdb.set_trace()`` by default it accepts no arguments.


Environment variable


The default implementation of ``sys.breakpointhook()`` consults a new
environment variable called ``PYTHONBREAKPOINT``.  This environment variable
can have various values:

* ``PYTHONBREAKPOINT=0`` disables debugging.  Specifically, with this value
  ``sys.breakpointhook()`` returns ``None`` immediately.

* ``PYTHONBREAKPOINT= `` (i.e. the empty string).  This is the same as not
  setting the environment variable at all, in which case ``pdb.set_trace()``
  is run as usual.

* ``PYTHONBREAKPOINT=some.importable.callable``.  In this case,
  ``sys.breakpointhook()`` imports the ``some.importable`` module and gets the
  ``callable`` object from the resulting module, which it then calls.  The
  value may be a string with no dots, in which case it names a built-in
  callable, e.g. ``PYTHONBREAKPOINT=int``.  (Guido has expressed the
  preference for normal Python dotted-paths, not setuptools-style entry point
  syntax [syntax]_.)

This environment variable allows external processes to control how breakpoints
are handled.  Some uses cases include:

* Completely disabling all accidental ``breakpoint()`` calls pushed to
  production.  This could be accomplished by setting ``PYTHONBREAKPOINT=0`` in
  the execution environment.  Another suggestion by reviewers of the PEP was
  to set ``PYTHONBREAKPOINT=sys.exit`` in this case.

* IDE integration with specialized debuggers for embedded execution.  The IDE
  would run the program in its debugging environment with ``PYTHONBREAKPOINT``
  set to their internal debugging hook.

``PYTHONBREAKPOINT`` is re-interpreted every time ``sys.breakpointhook()`` is
reached.  This allows processes to change its value during the execution of a
program and have ``breakpoint()`` respond to those changes.  It is not
considered a performance critical section since entering a debugger by
definition stops execution.  (Of note, the implementation fast-tracks the
``PYTHONBREAKPOINT=

Re: [Python-Dev] PEP 553, v3

2017-09-13 Thread Terry Reedy 

On 9/13/2017 10:12 PM, Barry Warsaw wrote:

Here’s an update to PEP 553, which makes $PYTHONBREAKPOINT a first class 
feature.  I’ve also updated PR #3355 with the implementation to match.


Looks pretty good to me.  Reading the PR eliminated my remaining 
uncertainties.


--
Terry Jan Reedy


___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v3 (new interpreters module)

2017-09-13 Thread Nick Coghlan 
On 14 September 2017 at 11:44, Eric Snow  wrote:
> I've updated PEP 554 in response to feedback.  (thanks all!)  There
> are a few unresolved points (some of them added to the Open Questions
> section), but the current PEP has changed enough that I wanted to get
> it out there first.
>
> Notably changed:
>
> * the API relative to object passing has changed somewhat drastically
> (hopefully simpler and easier to understand), replacing "FIFO" with
> "channel"
> * added an examples section
> * added an open questions section
> * added a rejected ideas section
> * added more items to the deferred functionality section
> * the rationale section has moved down below the examples
>
> Please let me know what you think.  I'm especially interested in
> feedback about the channels.  Thanks!

I like the new pipe-like channels API more than the previous named
FIFO approach :)

>send(obj):
>
>Send the object to the receiving end of the channel.  Wait until
>the object is received.  If the channel does not support the
>object then TypeError is raised.  Currently only bytes are
>supported.  If the channel has been closed then EOFError is
>raised.

I still expect any form of object sharing to hinder your
per-interpreter GIL efforts, so restricting the initial implementation
to memoryview-only seems more future-proof to me.


> Pre-populate an interpreter
> ---
>
> ::
>
>interp = interpreters.create()
>interp.run("""if True:
>import some_lib
>import an_expensive_module
>some_lib.set_up()
>""")
>wait_for_request()
>interp.run("""if True:
>some_lib.handle_request()
>""")

I find the "if True:"'s sprinkled through the examples distracting, so
I'd prefer either:

1. Using textwrap.dedent; or
2. Assigning the code to a module level attribute

::
   interp = interpreters.create()
   setup_code = """\
   import some_lib
   import an_expensive_module
   some_lib.set_up()
   """
   interp.run(setup_code)
   wait_for_request()

   handler_code = """\
   some_lib.handle_request()
   """
   interp.run(handler_code)

> Handling an exception
> -
>
> ::
>
>interp = interpreters.create()
>try:
>interp.run("""if True:
>raise KeyError
>""")
>except KeyError:
>print("got the error from the subinterpreter")

As with the message passing through channels, I think you'll really
want to minimise any kind of implicit object sharing that may
interfere with future efforts to make the GIL truly an *interpreter*
lock, rather than the global process lock that it is currently.

One possible way to approach that would be to make the low level run()
API a more Go-style API rather than a Python-style one, and have it
return a (result, err) 2-tuple. "err.raise()" would then translate the
foreign interpreter's exception into a local interpreter exception,
but the *traceback* for that exception would be entirely within the
current interpreter.

> About Subinterpreters
> =
>
> Shared data
> ---
>
> Subinterpreters are inherently isolated (with caveats explained below),
> in contrast to threads.  This enables `a different concurrency model
> `_ than is currently readily available in Python.
> `Communicating Sequential Processes`_ (CSP) is the prime example.
>
> A key component of this approach to concurrency is message passing.  So
> providing a message/object passing mechanism alongside ``Interpreter``
> is a fundamental requirement.  This proposal includes a basic mechanism
> upon which more complex machinery may be built.  That basic mechanism
> draws inspiration from pipes, queues, and CSP's channels. [fifo]_
>
> The key challenge here is that sharing objects between interpreters
> faces complexity due in part to CPython's current memory model.
> Furthermore, in this class of concurrency, the ideal is that objects
> only exist in one interpreter at a time.  However, this is not practical
> for Python so we initially constrain supported objects to ``bytes``.
> There are a number of strategies we may pursue in the future to expand
> supported objects and object sharing strategies.
>
> Note that the complexity of object sharing increases as subinterpreters
> become more isolated, e.g. after GIL removal.  So the mechanism for
> message passing needs to be carefully considered.  Keeping the API
> minimal and initially restricting the supported types helps us avoid
> further exposing any underlying complexity to Python users.
>
> To make this work, the mutable shared state will be managed by the
> Python runtime, not by any of the interpreters.  Initially we will
> support only one type of objects for shared state: the channels provided
> by ``create_channel()``.  Channels, in turn, will carefully manage
> passing objects between interpreters.

Interpreters themselves will also need to be shared objects, as:

- they all have access to "

Re: [Python-Dev] PEP 554 v3 (new interpreters module)

2017-09-13 Thread Yury Selivanov 
On Wed, Sep 13, 2017 at 11:56 PM, Nick Coghlan  wrote:
[..]
>>send(obj):
>>
>>Send the object to the receiving end of the channel.  Wait until
>>the object is received.  If the channel does not support the
>>object then TypeError is raised.  Currently only bytes are
>>supported.  If the channel has been closed then EOFError is
>>raised.
>
> I still expect any form of object sharing to hinder your
> per-interpreter GIL efforts, so restricting the initial implementation
> to memoryview-only seems more future-proof to me.

+1.  Working with memoryviews is as convenient as with bytes.

Yury
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v3 (new interpreters module)

2017-09-13 Thread Nathaniel Smith 
On Sep 13, 2017 9:01 PM, "Nick Coghlan"  wrote:

On 14 September 2017 at 11:44, Eric Snow 
wrote:
>send(obj):
>
>Send the object to the receiving end of the channel.  Wait until
>the object is received.  If the channel does not support the
>object then TypeError is raised.  Currently only bytes are
>supported.  If the channel has been closed then EOFError is
>raised.

I still expect any form of object sharing to hinder your
per-interpreter GIL efforts, so restricting the initial implementation
to memoryview-only seems more future-proof to me.


I don't get it. With bytes, you can either share objects or copy them and
the user can't tell the difference, so you can change your mind later if
you want. But memoryviews require some kind of cross-interpreter strong
reference to keep the underlying buffer object alive. So if you want to
minimize object sharing, surely bytes are more future-proof.


> Handling an exception
> -
>
> ::
>
>interp = interpreters.create()
>try:
>interp.run("""if True:
>raise KeyError
>""")
>except KeyError:
>print("got the error from the subinterpreter")

As with the message passing through channels, I think you'll really
want to minimise any kind of implicit object sharing that may
interfere with future efforts to make the GIL truly an *interpreter*
lock, rather than the global process lock that it is currently.

One possible way to approach that would be to make the low level run()
API a more Go-style API rather than a Python-style one, and have it
return a (result, err) 2-tuple. "err.raise()" would then translate the
foreign interpreter's exception into a local interpreter exception,
but the *traceback* for that exception would be entirely within the
current interpreter.


It would also be reasonable to simply not return any value/exception from
run() at all, or maybe just a bool for whether there was an unhandled
exception. Any high level API is going to be injecting code on both sides
of the interpreter boundary anyway, so it can do whatever exception and
traceback translation it wants to.


> Reseting __main__
> -
>
> As proposed, every call to ``Interpreter.run()`` will execute in the
> namespace of the interpreter's existing ``__main__`` module.  This means
> that data persists there between ``run()`` calls.  Sometimes this isn't
> desireable and you want to execute in a fresh ``__main__``.  Also,
> you don't necessarily want to leak objects there that you aren't using
> any more.
>
> Solutions include:
>
> * a ``create()`` arg to indicate resetting ``__main__`` after each
>   ``run`` call
> * an ``Interpreter.reset_main`` flag to support opting in or out
>   after the fact
> * an ``Interpreter.reset_main()`` method to opt in when desired
>
> This isn't a critical feature initially.  It can wait until later
> if desirable.

I was going to note that you can already do this:

interp.run("globals().clear()")

However, that turns out to clear *too* much, since it also clobbers
all the __dunder__ attributes that the interpreter needs in a code
execution environment.

Either way, if you added this, I think it would make more sense as an
"importlib.util.reset_globals()" operation, rather than have it be
something specific to subinterpreters.


This is another point where the API could reasonably say that if you want
clean namespaces then you should do that yourself (e.g. by setting up your
own globals dict and using it to execute any post-bootstrap code).

-n
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com