Re: [Python-Dev] PEP 550 v3

[Nick Coghlan]

On 19 August 2017 at 06:33, Yury Selivanov  wrote:
> Hi,
>
> This is a third iteration of the PEP.
>
> There was some really good feedback on python-ideas and
> the discussion thread became hard to follow again, so I decided to
> update the PEP only three days after I published the previous
> version.
>
> Summary of the changes can be found in the "Version History"
> section: https://www.python.org/dev/peps/pep-0550/#version-history
>
> There are a few open questions left, namely the terminology
> and design of ContextKey API.  On the former topic, I'm quite
> happy with the latest version: Execution Context, Logical
> Context, and Context Key.

Nice, I quite like this version of the naming scheme and the core
design in general.

While Guido has a point using the same noun for two different things
being somewhat confusing, I think the parallel here is the one between
the local scope and the lexical (nonlocal) scope for variable names -
just as your lexical scope is a nested stack of local scopes in outer
functions, your execution context is your current logical context plus
a nested stack of outer logical contexts.

> Generator Object Modifications
> ^^
>
> To achieve this, we make a small set of modifications to the
> generator object:
>
> * New ``__logical_context__`` attribute.  This attribute is readable
>   and writable for Python code.
>
> * When a generator object is instantiated its ``__logical_context__``
>   is initialized with an empty ``LogicalContext``.
>
> * Generator's ``.send()`` and ``.throw()`` methods are modified as
>   follows (in pseudo-C)::
>
> if gen.__logical_context__ is not NULL:
> tstate = PyThreadState_Get()
>
> tstate.execution_context.push(gen.__logical_context__)
>
> try:
> # Perform the actual `Generator.send()` or
> # `Generator.throw()` call.
> return gen.send(...)
> finally:
> gen.__logical_context__ = tstate.execution_context.pop()
> else:
> # Perform the actual `Generator.send()` or
> # `Generator.throw()` call.
> return gen.send(...)

I think this pseudo-code expansion includes a few holdovers from the
original visibly-immutable API design.

Given the changes since then, I think this would be clearer if the
first branch used sys.run_with_logical_context(), since the logical
context references at the Python layer now behave like shared mutable
objects, and the apparent immutability of
sys.run_with_execution_context() comes from injecting a fresh logical
context every time.

Also +1 to the new design considerations questions that explicitly
postpones consideration of any of my "What about..."" questions from
python-ideas to future PEPs.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Antoine Pitrou]

On Fri, 18 Aug 2017 16:33:27 -0400
Yury Selivanov  wrote:
> 
> There are a few open questions left, namely the terminology
> and design of ContextKey API.  On the former topic, I'm quite
> happy with the latest version: Execution Context, Logical
> Context, and Context Key.

I don't really like it.  "Logical Context" is vague (there are
lots of things called "context" in other libraries, so a bit of
specificity would help avoid confusion), and it's not clear what is
"logical" about it anyway.  "Local Context" actually seemed better to
me (as it reminded of threading.local() or the general notion of
thread-local storage).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Brett Cannon]

On Sat, Aug 19, 2017, 01:43 Antoine Pitrou  wrote:

> On Fri, 18 Aug 2017 16:33:27 -0400
> Yury Selivanov  wrote:
> >
> > There are a few open questions left, namely the terminology
> > and design of ContextKey API.  On the former topic, I'm quite
> > happy with the latest version: Execution Context, Logical
> > Context, and Context Key.
>
> I don't really like it.  "Logical Context" is vague (there are
> lots of things called "context" in other libraries, so a bit of
> specificity would help avoid confusion), and it's not clear what is
> "logical" about it anyway.  "Local Context" actually seemed better to
> me (as it reminded of threading.local() or the general notion of
> thread-local storage).
>
>
I have to admit that I didn't even pick up on that name change. I could go
either way.

I do appreciate dropping ContextItem, though, because "CI" makes me think
of continuous integration. And the overall shape of the API for public
consumption LGTM.

-brett


>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Ethan Furman]

On 08/19/2017 01:40 AM, Antoine Pitrou wrote:

On Fri, 18 Aug 2017 16:33:27 -0400 Yury Selivanov wrote:



There are a few open questions left, namely the terminology
and design of ContextKey API.  On the former topic, I'm quite
happy with the latest version: Execution Context, Logical
Context, and Context Key.


I don't really like it.  "Logical Context" is vague (there are
lots of things called "context" in other libraries, so a bit of
specificity would help avoid confusion), and it's not clear what is
"logical" about it anyway.  "Local Context" actually seemed better to
me (as it reminded of threading.local() or the general notion of
thread-local storage).


I am also not seeing the link between "logical" and "this local layer of environmental changes that won't be seen by 
those who called me".


Maybe ContextLayer?  Or marry the two and call it LocalContextLayer.

--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Guido van Rossum]

The way we came to "logical context" was via "logical thread (of control)",
which is distinct from OS thread. But I think we might need to search for
another term...

On Aug 19, 2017 11:56 AM, "Ethan Furman"  wrote:

> On 08/19/2017 01:40 AM, Antoine Pitrou wrote:
>
>> On Fri, 18 Aug 2017 16:33:27 -0400 Yury Selivanov wrote:
>>
>
> There are a few open questions left, namely the terminology
>>> and design of ContextKey API.  On the former topic, I'm quite
>>> happy with the latest version: Execution Context, Logical
>>> Context, and Context Key.
>>>
>>
>> I don't really like it.  "Logical Context" is vague (there are
>> lots of things called "context" in other libraries, so a bit of
>> specificity would help avoid confusion), and it's not clear what is
>> "logical" about it anyway.  "Local Context" actually seemed better to
>> me (as it reminded of threading.local() or the general notion of
>> thread-local storage).
>>
>
> I am also not seeing the link between "logical" and "this local layer of
> environmental changes that won't be seen by those who called me".
>
> Maybe ContextLayer?  Or marry the two and call it LocalContextLayer.
>
> --
> ~Ethan~
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Nick Coghlan]

On 20 August 2017 at 10:21, Guido van Rossum  wrote:
> The way we came to "logical context" was via "logical thread (of control)",
> which is distinct from OS thread. But I think we might need to search for
> another term...

Right. Framing it in pragmatic terms, the two entities that we're
attempting to name are:

1. The mutable storage that ContextKey.set() writes to
2. The dynamic context that ContextKey.get() queries

Right now, we're using ExecutionContext for the latter, and
LogicalContext for the former, and I can definitely see Antoine's
point that those names don't inherently convey any information about
which is which.

Personally, I still like the idea of moving ExecutionContext into the
"mutable storage" role, and then finding some other name for the stack
of execution contexts that ck.get() queries.

For example, if we named the latter after what it's *for*, we could
call it the DynamicQueryContext, and end up with the following
invocation functions:

# Replacing ExecutionContext in the current PEP
DynamicQueryContext
sys.get_dynamic_query_context()
sys.new_dynamic_query_context()
sys.run_with_dynamic_query_context()
# Suggests immutability -> good!
# Suggests connection to ck.get() -> good!

# Replacing LogicalContext in the current PEP
ExecutionContext
sys.new_execution_context()
sys.run_with_execution_context()
__execution_context__ attribute on generators (et al)
# Neutral on mutability/immutability
# Neutral on ck.set()/ck.get()

An alternative would be to dispense with the ExecutionContext name
entirely, and instead use DynamicWriteContext and DynamicQueryContext.
If we did that, I'd suggest omitting "dynamic" from the function and
attribute names (while keeping it on the types), and end up with:

# Replacing ExecutionContext in the current PEP
DynamicQueryContext
sys.get_query_context()
sys.new_query_context()
sys.run_with_query_context()
# Suggests immutability -> good!
# Suggests connection to ck.get() -> good!

# Replacing LogicalContext in the current PEP
DynamicWriteContext
sys.new_write_context()
sys.run_with_write_context()
__write_context__ attribute on generators (et al)
# Suggests mutability -> good!
# Suggests connection to ck.set() -> good!

In this variant, the phrase "execution context" could become a general
term that covered *all* of the active state that a running piece of
code has access to (the dynamic context,  thread locals, closure
variables, module globals, process globals, etc), rather than
referring to any particular runtime entity.

Cheers,
Nick.

P.S. Since we have LookupError (rather than QueryError) as the shared
base exception type for KeyError and IndexError, it would also be
entirely reasonable to replace "Query" in the above suggestions with
"Lookup" (DynamicLookupContext, sys.get_lookup_context(), etc). That
would also have the benefit of being less jargony, and more like
conversational English.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Ethan Furman]

On 08/19/2017 10:41 PM, Nick Coghlan wrote:

On 20 August 2017 at 10:21, Guido van Rossum wrote:



The way we came to "logical context" was via "logical thread (of control)",
which is distinct from OS thread. But I think we might need to search for
another term...


Right. Framing it in pragmatic terms, the two entities that we're
attempting to name are:

1. The mutable storage that ContextKey.set() writes to
2. The dynamic context that ContextKey.get() queries

Right now, we're using ExecutionContext for the latter, and
LogicalContext for the former, and I can definitely see Antoine's
point that those names don't inherently convey any information about
which is which.


> [snip]


 # Replacing ExecutionContext in the current PEP
 DynamicQueryContext
 sys.get_dynamic_query_context()
 sys.new_dynamic_query_context()
 sys.run_with_dynamic_query_context()
 # Suggests immutability -> good!
 # Suggests connection to ck.get() -> good!

 # Replacing LogicalContext in the current PEP
 ExecutionContext
 sys.new_execution_context()
 sys.run_with_execution_context()
 __execution_context__ attribute on generators (et al)
 # Neutral on mutability/immutability
 # Neutral on ck.set()/ck.get()



[snippety snip]



 # Replacing ExecutionContext in the current PEP
 DynamicQueryContext
 sys.get_query_context()
 sys.new_query_context()
 sys.run_with_query_context()
 # Suggests immutability -> good!
 # Suggests connection to ck.get() -> good!

 # Replacing LogicalContext in the current PEP
 DynamicWriteContext
 sys.new_write_context()
 sys.run_with_write_context()
 __write_context__ attribute on generators (et al)
 # Suggests mutability -> good!
 # Suggests connection to ck.set() -> good!


This is just getting more confusing for me.

Going back to Yury's original names for now...

Relating this naming problem back to globals() and locals(), the correlation works okay for locals/LocalContext, but 
breaks down at the globals() level because globals() is a specific set of variables -- namely, module-level assignments, 
while ExecutionContext would be the equivalent of globals, nonlocals, and locals all together.


A more accurate name for ExecutionContext might be ParentContext, but that would imply that the LocalContext is not 
included, and it is (if I finally understand what's going on, of course).


So I like ExecutionContext for the stack of WhateverWeCallTheOtherContext contexts.  But what do we call it?  Again, if 
I understand what's going on, a normal, threadless, non-async, generator-bereft, plain vanilla Python program is going 
to have only one LocalContext no matter how many nor how deep the function call chain goes -- so in that sense Local 
isn't really the best word, but Context all by itself is /really/ unhelpful, and Local does imply "the most current 
Context Layer".


Of all the names proposed so far, I think LocalContext is the best reminder of the thing that CK.key writes to.  For the 
piled layers of LocalContexts that CK.key.get searches through, either ExecutionContext or perhaps ContextEnvironment or 
even ContextStack works for me (the stack portion not being an implementation detail, but a promise of how it 
effectively works).


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Antoine Pitrou]

On Sat, 19 Aug 2017 17:21:03 -0700
Guido van Rossum  wrote:
> The way we came to "logical context" was via "logical thread (of control)",
> which is distinct from OS thread. But I think we might need to search for
> another term...

Perhaps "task context"?  A "task" might be a logical thread, OS thread,
or anything else that deserves a distinct set of implicit parameters.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[francismb]

Hi Yury,

On 08/18/2017 10:33 PM, Yury Selivanov wrote:
> * ``.get()`` method: return the current EC value for the context key.
>   Context keys return ``None`` when the key is missing, so the method
>   never fails.
Is the difference between `Key not found` and `value is None` important
here?

Thanks,
--francis
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Brett Cannon]

On Sun, Aug 20, 2017, 03:08 Antoine Pitrou  wrote:

> On Sat, 19 Aug 2017 17:21:03 -0700
> Guido van Rossum  wrote:
> > The way we came to "logical context" was via "logical thread (of
> control)",
> > which is distinct from OS thread. But I think we might need to search for
> > another term...
>
> Perhaps "task context"?  A "task" might be a logical thread, OS thread,
> or anything else that deserves a distinct set of implicit parameters.
>

Maybe this is skirting too loose to the dynamic scoping, but maybe
ContextFrame? This does start to line up with frames of execution which I
know is a bit low-level, but then again most people will never need to know
about this corner of Python.

-brett


> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Guido van Rossum]

>
> On Sun, Aug 20, 2017, 03:08 Antoine Pitrou  wrote:
>
>> On Sat, 19 Aug 2017 17:21:03 -0700
>> Guido van Rossum  wrote:
>> > The way we came to "logical context" was via "logical thread (of
>> control)",
>> > which is distinct from OS thread. But I think we might need to search
>> for
>> > another term...
>>
>> Perhaps "task context"?  A "task" might be a logical thread, OS thread,
>> or anything else that deserves a distinct set of implicit parameters.
>>
>
I think you're on to something here, though I hesitate to use "task"
because asyncio.Task is a specific implementation of it.

On Sun, Aug 20, 2017 at 7:04 PM, Brett Cannon  wrote:

> Maybe this is skirting too loose to the dynamic scoping, but maybe
> ContextFrame? This does start to line up with frames of execution which I
> know is a bit low-level, but then again most people will never need to know
> about this corner of Python.
>

I've been thinking that the missing link here may be the execution stack. A
logical thread (LT) has a "logical stack" (LS). While a thread of control
is a fairly fuzzy concept (once you include things that aren't OS threads),
an execution stack is a concrete object, even if not all logical threads
represent their execution stack in the same way. For example, a suspended
asyncio Task has a stack that is represented by a series of stack frames
linked together by await (or yield from), and greenlet apparently uses a
different representation again (their term is micro-thread -- maybe we
could also do something with that?).

Here's the result of some more thinking about this PEP that I've been doing
while writing and rewriting this message (with a surprising ending).

Let's say that the details of scheduling an LT and managing its mapping
onto an LS is defined by a "framework". In this terminology, OS threads are
a framework, as are generators, asyncio, and greenlet. There are
potentially many different such frameworks. (Some others include Twisted,
Tornado and concurrent.futures.ThreadPoolExecutor.)

The PEP's big idea is to recognize that there are also many different,
non-framework, libraries (e.g. Decimal or Flask) that need to associate
some data with an LT. The PEP therefore proposes APIs that allow libraries
to do this without caring about what framework is managing the LT, and vice
versa (the framework doesn't have to care about how libraries use the
per-LT data).

The proposed APIs uses two sets of concepts: one set for the framework and
one for the library.

The library-facing API is simple: create a ContextKey (CK) instance as a
global variable in the library, and use its get() and set() methods to
access and manipulate the data for that library associated with the current
logical thread (LT). Its role is similar to threading.local(), although the
API and implementation are completely different, and threading.local() is
tied to a specific framework (OS threads).

For frameworks the API is more complicated. There are two new classes,
LogicalContext (LC) and ExecutionContext (EC). The PEP gives pseudo code
suggesting that LC is/contains a dict (whose items are (CK, value) pairs)
and an EC is/contains a list of LCs. But in actuality that's only one
possible implementation (and not the one proposed for CPython). The key
idea is rather that a framework needs to be able to take the data
associated with one LT and clone it as the starting point for the data
associated for a new LT. This cloning is done by
sys.get_execution_context(), and the PEP proposes to use a Hash Array
Mapped Trie (HAMT) as the basis for the implementation of LC and EC, to
make this cloning fast. IIUC it needs to be fast to match the speed with
which many frameworks create and destroy their LTs.

The PEP proposes a bunch of new functions in sys for frameworks to
manipulate LCs and ECs and their association with the current OS-level
thread. Note that OS threads are important here because in the end all
frameworks build on top of them.

Honestly I'm not sure we need the distinction between LC and EC. If you
read carefully some of the given example code seems to confuse them. If we
could get away with only a single framework-facing concept, I would be
happy calling it ExecutionContext.

(Another critique of the proposal I have is that it adds too many
similarly-named functions to sys. But this email is already too long and I
need to go to bed.)

-- 
--Guido van Rossum (python.org/~guido )
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Nick Coghlan]

On 21 August 2017 at 15:03, Guido van Rossum  wrote:
> Honestly I'm not sure we need the distinction between LC and EC. If you read
> carefully some of the given example code seems to confuse them. If we could
> get away with only a single framework-facing concept, I would be happy
> calling it ExecutionContext.

Unfortunately, I don't think we can, and that's why I tried to reframe
the discussion in terms of "Where ContextKey.set() writes to" and
"Where ContextKey.get() looks things up".

Consider the following toy generator:

def tracking_gen():
start_tracking_iterations()
while True:
tally_iteration()
yield

task_id = ContextKey("task_id")
iter_counter = ContextKey("iter_counter")

def start_tracking_iterations():
iter_counter.set(collection.Counter())

def tally_iteration():
current_task = task_id.get() # Set elsewhere
iter_counter.get()[current_task] += 1

Now, this isn't a very *sensible* generator (since it could just use a
regular object instance for tracking instead of a context variable),
but nevertheless, it's one that we would expect to work, and it's one
that we would expect to exhibit the following properties:

1. When tally_iteration() calls task_id.get(), we expect that to be
resolved in the context calling next() on the instance, *not* the
context where the generator was first created
2. When tally_iteration() calls iter_counter.get(), we expect that to
be resolved in the same context where start_tracking_iterations()
called iter_counter.set()

This has consequences for the design in the PEP:

* what we want to capture at generator creation time is the context
where writes will happen, and we also want that to be the innermost
context used for lookups
* other than that innermost context, we want everything else to be dynamic
* this means that "mutable context saved on the generator" and "entire
dynamic context visible when the generator runs" aren't the same thing

And hence the introduction of the LocalContext/LogicalContext
terminology for the former, and the ExecutionContext terminology for
the latter.

It's also where the analogy with ChainMap came from (although I don't
think this has made it into the PEP itself):

* LogicalContext is the equivalent of the individual mappings
* ExecutionContext is the equivalent of ChainMap
* ContextKey.get() replaces ChainMap.__getitem__
* ContextKey.set(value) replaces ChainMap.__setitem__
* ContextKey.set(None) replaces ChainMap.__delitem__

While the context is defined conceptually as a nested chain of
key:value mappings, we avoid using the mapping syntax because of the
way the values can shift dynamically out from under you based on who
called you - while the ChainMap analogy is hopefully helpful to
understanding, we don't want people taking it too literally or things
will become more confusing rather than less.

Despite that risk, taking the analogy further is where the
DynamicWriteContext + DynamicLookupContext terminology idea came from:

* like ChainMap.new_child(), adjusting the DynamicWriteContext changes
what ck.set() affects, and also sets the innermost context for
ck.get()
* like using a different ChainMap, adjusting the DynamicLookupContext
changes what ck.get() can see (unlike ChainMap, it also isolates
ck.set() by default)

I'll also note that the first iteration of the PEP didn't really make
this distinction, and it caused a problem that Nathaniel pointed out:
generators would "snapshot" their entire dynamic context when first
created, and then never adjust it for external changes between
iterations. This meant that if you adjusted something like the decimal
context outside the generator after creating it, it would ignore those
changes - instead of having the problem of changes inside the
generator leaking out, we instead had the problem of changes outside
the generator *not* making their way in, even if you wanted them to.

Due to that heritage, fixing some of the examples could easily have
been missed in the v2 rewrite that introduced the distinction between
the two kinds of context.

> (Another critique of the proposal I have is that it adds too many
> similarly-named functions to sys. But this email is already too long and I
> need to go to bed.)

If it helps any, one of the ideas that has come up is to put all of
the proposed context manipulation APIs in contextlib rather than in
sys, and I think that's a reasonable idea (I don't think any of us
actually like the notion of adding that many new subsystem specific
APIs directly to sys).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Guido van Rossum]

On Mon, Aug 21, 2017 at 7:12 AM, Nick Coghlan  wrote:

> On 21 August 2017 at 15:03, Guido van Rossum  wrote:
> > Honestly I'm not sure we need the distinction between LC and EC. If you
> read
> > carefully some of the given example code seems to confuse them. If we
> could
> > get away with only a single framework-facing concept, I would be happy
> > calling it ExecutionContext.
>
> Unfortunately, I don't think we can, and that's why I tried to reframe
> the discussion in terms of "Where ContextKey.set() writes to" and
> "Where ContextKey.get() looks things up".
>
> Consider the following toy generator:
>
> def tracking_gen():
> start_tracking_iterations()
> while True:
> tally_iteration()
> yield
>
> task_id = ContextKey("task_id")
> iter_counter = ContextKey("iter_counter")
>
> def start_tracking_iterations():
> iter_counter.set(collection.Counter())
>
> def tally_iteration():
> current_task = task_id.get() # Set elsewhere
> iter_counter.get()[current_task] += 1
>
> Now, this isn't a very *sensible* generator (since it could just use a
> regular object instance for tracking instead of a context variable),
> but nevertheless, it's one that we would expect to work, and it's one
> that we would expect to exhibit the following properties:
>
> 1. When tally_iteration() calls task_id.get(), we expect that to be
> resolved in the context calling next() on the instance, *not* the
> context where the generator was first created
> 2. When tally_iteration() calls iter_counter.get(), we expect that to
> be resolved in the same context where start_tracking_iterations()
> called iter_counter.set()
>
> This has consequences for the design in the PEP:
>
> * what we want to capture at generator creation time is the context
> where writes will happen, and we also want that to be the innermost
> context used for lookups
> * other than that innermost context, we want everything else to be dynamic
> * this means that "mutable context saved on the generator" and "entire
> dynamic context visible when the generator runs" aren't the same thing
>
> And hence the introduction of the LocalContext/LogicalContext
> terminology for the former, and the ExecutionContext terminology for
> the latter.
>

OK, this is a sensible explanation. I think the PEP would benefit from
including some version of it early on (though perhaps shortened a bit).


> It's also where the analogy with ChainMap came from (although I don't
> think this has made it into the PEP itself):
>
> * LogicalContext is the equivalent of the individual mappings
> * ExecutionContext is the equivalent of ChainMap
> * ContextKey.get() replaces ChainMap.__getitem__
> * ContextKey.set(value) replaces ChainMap.__setitem__
> * ContextKey.set(None) replaces ChainMap.__delitem__
>
> While the context is defined conceptually as a nested chain of
> key:value mappings, we avoid using the mapping syntax because of the
> way the values can shift dynamically out from under you based on who
> called you - while the ChainMap analogy is hopefully helpful to
> understanding, we don't want people taking it too literally or things
> will become more confusing rather than less.
>

Agreed. However now I am confused as to how the HAMT fits in. Yury says
somewhere that the HAMT will be used for the EC and then cloning the EC is
just returning a pointer to the same EC. But even if I interpret that as
making a new EC containing a pointer to the same underlying HAMT, I don't
see how that will preserve the semantics that different logical threads,
running interleaved (like different generators being pumped alternatingly),
will see updates to LCs that are lower on the stack of LCs in the EC. (I
see this with the stack-of-dicts version, but not with the immutable HAMT
inplementation.)


> Despite that risk, taking the analogy further is where the
> DynamicWriteContext + DynamicLookupContext terminology idea came from:
>
> * like ChainMap.new_child(), adjusting the DynamicWriteContext changes
> what ck.set() affects, and also sets the innermost context for
> ck.get()
> * like using a different ChainMap, adjusting the DynamicLookupContext
> changes what ck.get() can see (unlike ChainMap, it also isolates
> ck.set() by default)
>

Here I'm lost again. In the PEP's pseudo code, your first bullet seems to
be the operation "push a new LC on the stack of the current EC". Does the
second bullet just mean "switch to a different EC"?


> I'll also note that the first iteration of the PEP didn't really make
> this distinction, and it caused a problem that Nathaniel pointed out:
> generators would "snapshot" their entire dynamic context when first
> created, and then never adjust it for external changes between
> iterations. This meant that if you adjusted something like the decimal
> context outside the generator after creating it, it would ignore those
> changes - instead of having the problem of changes inside the
> generator 

Re: [Python-Dev] PEP 550 v3

[Yury Selivanov]

On Mon, Aug 21, 2017 at 3:10 PM, Guido van Rossum  wrote:
[..]
> Agreed. However now I am confused as to how the HAMT fits in. Yury says
> somewhere that the HAMT will be used for the EC and then cloning the EC is
> just returning a pointer to the same EC. But even if I interpret that as
> making a new EC containing a pointer to the same underlying HAMT, I don't
> see how that will preserve the semantics that different logical threads,
> running interleaved (like different generators being pumped alternatingly),
> will see updates to LCs that are lower on the stack of LCs in the EC. (I see
> this with the stack-of-dicts version, but not with the immutable HAMT
> inplementation.)

Few important things (using the current PEP 550 terminology):

* ExecutionContext is a *dynamic* stack of LogicalContexts.
* LCs do not reference other LCs.
* ContextKey.set() can only modify the *top* LC in the stack.

If LC is a mutable mapping:

 # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]

 a.set(c)
 #LC4 = EC.top()
 #LC4[a] = c

 # EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})]

If LC are implemented with immutable mappings:

 # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]

 a.set(c)
 #LC4 = EC.pop()
 #LC4_1 = LC4.copy()
 #LC4_1[a] = c
 #EC.push(LC4_1)

 # EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})]

Any code that uses EC will not see any difference, because it can only
work with the top LC.

Back to generators. Generators have their own empty LCs when created
to store their *local* EC modifications.

When a generator is *being* iterated, it pushes its LC to the EC. When
the iteration step is finished, it pops its LC from the EC.  If you
have nested generators, they will dynamically build a stack of their
LCs while they are iterated.

Therefore, generators *naturally* control the stack of EC.  We can't
execute two generators simultaneously in one thread (we can only
iterate them one by one), so the top LC always belongs to the current
generator that is being iterated:

def nested_gen():
# EC = [outer_LC, gen1_LC, nested_gen_LC]
yield
# EC = [outer_LC, gen1_LC, nested_gen_LC]
yield

def gen1():
# EC = [outer_LC, gen1_LC]
n = nested_gen()
yield
# EC = [outer_LC, gen1_LC]
next(n)
# EC = [outer_LC, gen1_LC]
yield
next(n)
# EC = [outer_LC, gen1_LC]

def gen2():
# EC = [outer_LC, gen2_LC]
yield
# EC = [outer_LC, gen2_LC]
yield

g1 = gen1()
g2 = gen2()

next(g1)
next(g2)
next(g1)
next(g2)

HAMT is a way to efficiently implement immutable mappings with O(log32
N) set operation, that's it.  If we implement immutable mappings using
regular dicts and copy, set() would be O(log N).

[..]
>>
>> I'll also note that the first iteration of the PEP didn't really make
>> this distinction, and it caused a problem that Nathaniel pointed out:
>> generators would "snapshot" their entire dynamic context when first
>> created, and then never adjust it for external changes between
>> iterations. This meant that if you adjusted something like the decimal
>> context outside the generator after creating it, it would ignore those
>> changes - instead of having the problem of changes inside the
>> generator leaking out, we instead had the problem of changes outside
>> the generator *not* making their way in, even if you wanted them to.
>
>
> OK, this really needs to be made very clear early in the PEP. Maybe this
> final sentence provides the key requirement: changes outside the generator
> should make it into the generator when next() is invoked, unless the
> generator itself has made an override; but changes inside the generator
> should not leak out through next().

It's covered here with two examples:
https://www.python.org/dev/peps/pep-0550/#ec-semantics-for-generators

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Yury Selivanov]

On Sat, Aug 19, 2017 at 4:17 AM, Nick Coghlan  wrote:
[..]
>>
>> * Generator's ``.send()`` and ``.throw()`` methods are modified as
>>   follows (in pseudo-C)::
>>
>> if gen.__logical_context__ is not NULL:
>> tstate = PyThreadState_Get()
>>
>> tstate.execution_context.push(gen.__logical_context__)
>>
>> try:
>> # Perform the actual `Generator.send()` or
>> # `Generator.throw()` call.
>> return gen.send(...)
>> finally:
>> gen.__logical_context__ = tstate.execution_context.pop()
>> else:
>> # Perform the actual `Generator.send()` or
>> # `Generator.throw()` call.
>> return gen.send(...)
>
> I think this pseudo-code expansion includes a few holdovers from the
> original visibly-immutable API design.
>
> Given the changes since then, I think this would be clearer if the
> first branch used sys.run_with_logical_context(), since the logical
> context references at the Python layer now behave like shared mutable
> objects, and the apparent immutability of
> sys.run_with_execution_context() comes from injecting a fresh logical
> context every time.

This is a good idea, I like it!  It will indeed simplify the explanation.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Koos Zevenhoven]

On Mon, Aug 21, 2017 at 5:12 PM, Nick Coghlan  wrote:

> On 21 August 2017 at 15:03, Guido van Rossum  wrote:
> > Honestly I'm not sure we need the distinction between LC and EC. If you
> read
> > carefully some of the given example code seems to confuse them. If we
> could
> > get away with only a single framework-facing concept, I would be happy
> > calling it ExecutionContext.
>
> Unfortunately, I don't think we can, and that's why I tried to reframe
> the discussion in terms of "Where ContextKey.set() writes to" and
> "Where ContextKey.get() looks things up".
>
> Consider the following toy generator:
>
> def tracking_gen():
> start_tracking_iterations()
> while True:
> tally_iteration()
> yield
>
> task_id = ContextKey("task_id")
> iter_counter = ContextKey("iter_counter")
>
> def start_tracking_iterations():
> iter_counter.set(collection.Counter())
>
> def tally_iteration():
> current_task = task_id.get() # Set elsewhere
> iter_counter.get()[current_task] += 1
>
> Now, this isn't a very *sensible* generator (since it could just use a
> regular object instance for tracking instead of a context variable),
> but nevertheless, it's one that we would expect to work, and it's one
> that we would expect to exhibit the following properties:
>
> 1. When tally_iteration() calls task_id.get(), we expect that to be
> resolved in the context calling next() on the instance, *not* the
> context where the generator was first created
> 2. When tally_iteration() calls iter_counter.get(), we expect that to
> be resolved in the same context where start_tracking_iterations()
> called iter_counter.set()
>
> This has consequences for the design in the PEP:
>
> * what we want to capture at generator creation time is the context
> where writes will happen, and we also want that to be the innermost
> context used for lookups
>

​I don't get it. How is this a consequence of the above two points? And why
do we need to capture something (a "context") at generator creation time?

​-- Koos​



> * other than that innermost context, we want everything else to be dynamic
> * this means that "mutable context saved on the generator" and "entire
> dynamic context visible when the generator runs" aren't the same thing
>
> And hence the introduction of the LocalContext/LogicalContext
> terminology for the former, and the ExecutionContext terminology for
> the latter.
>
>
> ​[...]​




-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Yury Selivanov]

On Mon, Aug 21, 2017 at 5:14 PM, Koos Zevenhoven  wrote:
[..]
>> This has consequences for the design in the PEP:
>>
>> * what we want to capture at generator creation time is the context
>> where writes will happen, and we also want that to be the innermost
>> context used for lookups
>
>
> I don't get it. How is this a consequence of the above two points? And why
> do we need to capture something (a "context") at generator creation time?
>

We don't need to "capture" anything when a generator is created (it
was something that PEP 550 version 1 was doing).

In the current version of the PEP, generators are initialized with an
empty LogicalContext.  When they are being iterated (started or
resumed), their LogicalContext is pushed to the EC.  When the
iteration is stopped (or paused), they pop their LC from the EC.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Koos Zevenhoven]

On Tue, Aug 22, 2017 at 12:25 AM, Yury Selivanov 
wrote:

> On Mon, Aug 21, 2017 at 5:14 PM, Koos Zevenhoven 
> wrote:
> [..]
> >> This has consequences for the design in the PEP:
> >>
> >> * what we want to capture at generator creation time is the context
> >> where writes will happen, and we also want that to be the innermost
> >> context used for lookups
> >
> >
> > I don't get it. How is this a consequence of the above two points? And
> why
> > do we need to capture something (a "context") at generator creation time?
> >
>
> We don't need to "capture" anything when a generator is created (it
> was something that PEP 550 version 1 was doing).
>
>
​Ok, good.​



> In the current version of the PEP, generators are initialized with an
> empty LogicalContext.  When they are being iterated (started or
> resumed), their LogicalContext is pushed to the EC.  When the
> iteration is stopped (or paused), they pop their LC from the EC.
>
>
Another quick one before I go: Do we really need to push and pop a LC on
each next() call​, even if it most likely will never be touched?

-- Koos

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Yury Selivanov]

On Mon, Aug 21, 2017 at 5:39 PM, Koos Zevenhoven  wrote:
[..]
>> In the current version of the PEP, generators are initialized with an
>> empty LogicalContext.  When they are being iterated (started or
>> resumed), their LogicalContext is pushed to the EC.  When the
>> iteration is stopped (or paused), they pop their LC from the EC.
>>
>
> Another quick one before I go: Do we really need to push and pop a LC on
> each next() call, even if it most likely will never be touched?

Yes, otherwise it will be hard to maintain the consistency of the stack.

There will be an optimization: if the LC is empty, we will push NULL
to the stack, thus avoiding the cost of allocating an object.

I measured the overhead -- generators will become 0.5-1% slower in
microbenchmarks, but only when they do pretty much nothing. If a
generator contains more Python code than a bare "yield" expression,
the overhead will be harder to detect.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Guido van Rossum]

On Mon, Aug 21, 2017 at 12:50 PM, Yury Selivanov 
wrote:

> Few important things (using the current PEP 550 terminology):
>
> * ExecutionContext is a *dynamic* stack of LogicalContexts.
> * LCs do not reference other LCs.
> * ContextKey.set() can only modify the *top* LC in the stack.
>
> If LC is a mutable mapping:
>
>  # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
>
>  a.set(c)
>  #LC4 = EC.top()
>  #LC4[a] = c
>
>  # EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})]
>
> If LC are implemented with immutable mappings:
>
>  # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
>
>  a.set(c)
>  #LC4 = EC.pop()
>  #LC4_1 = LC4.copy()
>  #LC4_1[a] = c
>  #EC.push(LC4_1)
>
>  # EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})]
>
> Any code that uses EC will not see any difference, because it can only
> work with the top LC.
>

OK, good. This makes more sense, especially if I read "the EC" as shorthand
for the EC stored in the current thread's per-thread state. The immutable
mapping (if used) is used for the LC, not for the EC, and in this case
cloning an EC would simply make a shallow copy of its underlying list --
whereas without the immutable mapping, cloning the EC would also require
making shallow copies of each LC. And I guess the linked-list
implementation (Approach #3 in the PEP) makes EC cloning an O(1) operation.

Note that there is a lot of hand-waving and shorthand in this explanation,
but I think I finally follow the design. It is going to be a big task to
write this up in a didactic way -- the current PEP needs a fair amount of
help in that sense. (If you want to become a better writer, I've recently
enjoyed reading Steven Pinker's *The Sense of Style*: The Thinking Person's
Guide to Writing in the 21st Century. Amongst other fascinating topics, it
explains why so often what we think is clearly written can cause so much
confusion.)


> Back to generators. Generators have their own empty LCs when created
> to store their *local* EC modifications.
>
> When a generator is *being* iterated, it pushes its LC to the EC. When
> the iteration step is finished, it pops its LC from the EC.  If you
> have nested generators, they will dynamically build a stack of their
> LCs while they are iterated.
>
> Therefore, generators *naturally* control the stack of EC.  We can't
> execute two generators simultaneously in one thread (we can only
> iterate them one by one), so the top LC always belongs to the current
> generator that is being iterated:
>
> def nested_gen():
> # EC = [outer_LC, gen1_LC, nested_gen_LC]
> yield
> # EC = [outer_LC, gen1_LC, nested_gen_LC]
> yield
>
> def gen1():
> # EC = [outer_LC, gen1_LC]
> n = nested_gen()
> yield
> # EC = [outer_LC, gen1_LC]
> next(n)
> # EC = [outer_LC, gen1_LC]
> yield
> next(n)
> # EC = [outer_LC, gen1_LC]
>
> def gen2():
> # EC = [outer_LC, gen2_LC]
> yield
> # EC = [outer_LC, gen2_LC]
> yield
>
> g1 = gen1()
> g2 = gen2()
>
> next(g1)
> next(g2)
> next(g1)
> next(g2)
>

This, combined with your later clarification:

> In the current version of the PEP, generators are initialized with an
> empty LogicalContext.  When they are being iterated (started or
> resumed), their LogicalContext is pushed to the EC.  When the
> iteration is stopped (or paused), they pop their LC from the EC.

makes it clear how the proposal works for generators. There's an important
piece that I hadn't figured out from Nick's generator example, because I
had mistakenly assumed that something *would* be captured at generator
create time. It's a reasonable mistake to make, I think -- the design space
here is just huge and there are many variations that don't affect typical
code but do differ in edge cases. Your clear statement "nothing needs to be
captured" is helpful to avoid this misunderstanding.


> HAMT is a way to efficiently implement immutable mappings with O(log32
> N) set operation, that's it.  If we implement immutable mappings using
> regular dicts and copy, set() would be O(log N).
>

This sounds like abuse of the O() notation. Mathematically O(log N) and
O(log32 N) surely must be equivalent, since log32 N is just K*(log N) for
some constant K (about 0.288539), and the constant disappears in the O(),
as O(K*f(N)) and O(f(N)) are equivalent. Now, I'm happy to hear that a
HAMT-based implementation is faster than a dict+copy-based implementation,
but I don't think your use of O() makes sense here.


> [..]
> >>
> >> I'll also note that the first iteration of the PEP didn't really make
> >> this distinction, and it caused a problem that Nathaniel pointed out:
> >> generators would "snapshot" their entire dynamic context when first
> >> created, and then never adjust it for external changes between
> >> iterations. This meant that if you adjusted something like the

Re: [Python-Dev] PEP 550 v3

[Yury Selivanov]

On Mon, Aug 21, 2017 at 8:06 PM, Guido van Rossum  wrote:
> On Mon, Aug 21, 2017 at 12:50 PM, Yury Selivanov 
> wrote:
>>
>> Few important things (using the current PEP 550 terminology):
>>
>> * ExecutionContext is a *dynamic* stack of LogicalContexts.
>> * LCs do not reference other LCs.
>> * ContextKey.set() can only modify the *top* LC in the stack.
>>
>> If LC is a mutable mapping:
>>
>>  # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
>>
>>  a.set(c)
>>  #LC4 = EC.top()
>>  #LC4[a] = c
>>
>>  # EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})]
>>
>> If LC are implemented with immutable mappings:
>>
>>  # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
>>
>>  a.set(c)
>>  #LC4 = EC.pop()
>>  #LC4_1 = LC4.copy()
>>  #LC4_1[a] = c
>>  #EC.push(LC4_1)
>>
>>  # EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})]
>>
>> Any code that uses EC will not see any difference, because it can only
>> work with the top LC.
>
>
> OK, good. This makes more sense, especially if I read "the EC" as shorthand
> for the EC stored in the current thread's per-thread state.

That's exactly what I meant by "the EC".

> The immutable
> mapping (if used) is used for the LC, not for the EC, and in this case
> cloning an EC would simply make a shallow copy of its underlying list --
> whereas without the immutable mapping, cloning the EC would also require
> making shallow copies of each LC. And I guess the linked-list implementation
> (Approach #3 in the PEP) makes EC cloning an O(1) operation.

All correct.

>
> Note that there is a lot of hand-waving and shorthand in this explanation,
> but I think I finally follow the design. It is going to be a big task to
> write this up in a didactic way -- the current PEP needs a fair amount of
> help in that sense.

Elvis Pranskevichus (our current What's New editor and my colleague)
offered me to help with the PEP. He's now working on a partial rewrite.

I've been working on this PEP for about a month now and at this point
it makes it difficult for me to dump this knowledge in a nice and
readable way (in any language that I know, FWIW).

> (If you want to become a better writer, I've recently
> enjoyed reading Steven Pinker's The Sense of Style: The Thinking Person's
> Guide to Writing in the 21st Century. Amongst other fascinating topics, it
> explains why so often what we think is clearly written can cause so much
> confusion.)

Will definitely check it out, thank you!

>
>>
>> Back to generators. Generators have their own empty LCs when created
>> to store their *local* EC modifications.
>>
>> When a generator is *being* iterated, it pushes its LC to the EC. When
>> the iteration step is finished, it pops its LC from the EC.  If you
>> have nested generators, they will dynamically build a stack of their
>> LCs while they are iterated.
>>
>> Therefore, generators *naturally* control the stack of EC.  We can't
>> execute two generators simultaneously in one thread (we can only
>> iterate them one by one), so the top LC always belongs to the current
>> generator that is being iterated:
>>
>> def nested_gen():
>> # EC = [outer_LC, gen1_LC, nested_gen_LC]
>> yield
>> # EC = [outer_LC, gen1_LC, nested_gen_LC]
>> yield
>>
>> def gen1():
>> # EC = [outer_LC, gen1_LC]
>> n = nested_gen()
>> yield
>> # EC = [outer_LC, gen1_LC]
>> next(n)
>> # EC = [outer_LC, gen1_LC]
>> yield
>> next(n)
>> # EC = [outer_LC, gen1_LC]
>>
>> def gen2():
>> # EC = [outer_LC, gen2_LC]
>> yield
>> # EC = [outer_LC, gen2_LC]
>> yield
>>
>> g1 = gen1()
>> g2 = gen2()
>>
>> next(g1)
>> next(g2)
>> next(g1)
>> next(g2)
>
>
> This, combined with your later clarification:
>
>> In the current version of the PEP, generators are initialized with an
>> empty LogicalContext.  When they are being iterated (started or
>> resumed), their LogicalContext is pushed to the EC.  When the
>> iteration is stopped (or paused), they pop their LC from the EC.
>
> makes it clear how the proposal works for generators. There's an important
> piece that I hadn't figured out from Nick's generator example, because I had
> mistakenly assumed that something *would* be captured at generator create
> time. It's a reasonable mistake to make,

Yeah, it is very subtle.

>
>>
>> HAMT is a way to efficiently implement immutable mappings with O(log32
>> N) set operation, that's it.  If we implement immutable mappings using
>> regular dicts and copy, set() would be O(log N).
>
>
> This sounds like abuse of the O() notation. Mathematically O(log N) and
> O(log32 N) surely must be equivalent, since log32 N is just K*(log N) for
> some constant K (about 0.288539), and the constant disappears in the O(), as
> O(K*f(N)) and O(f(N)) are equivalent. Now, I'm happy to hear that a
> HAMT-based implementation is faster than a dict+copy-based imple

Re: [Python-Dev] PEP 550 v3

[Yury Selivanov]

On Mon, Aug 21, 2017 at 8:06 PM, Guido van Rossum  wrote:
[..]
>> > OK, this really needs to be made very clear early in the PEP. Maybe this
>> > final sentence provides the key requirement: changes outside the
>> > generator
>> > should make it into the generator when next() is invoked, unless the
>> > generator itself has made an override; but changes inside the generator
>> > should not leak out through next().
>>
>> It's covered here with two examples:
>> https://www.python.org/dev/peps/pep-0550/#ec-semantics-for-generators
>
>
> I think what's missing is the fact that this is one of the key motivating
> reasons for the design (starting with v2 of the PEP). When I encountered
> that section I just skimmed it, assuming it was mostly just showing how to
> apply the given semantics to generators. I also note some issues with the
> use of tense here -- it's a bit confusing to follow which parts of the text
> refer to defects of the current (pre-PEP) situation and which parts refer to
> how the proposal would solve these defects.

I see.  The proposal always uses present tense to describe things it
adds, and I now see that this is indeed very confusing.  This needs to
be fixed.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Greg Ewing]

Ethan Furman wrote:
So I like ExecutionContext for the stack of 
WhateverWeCallTheOtherContext contexts.  But what do we call it?


How about ExecutionContextFrame, by analogy with stack/stack frame.

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Nick Coghlan]

On 22 August 2017 at 09:39, Greg Ewing  wrote:
> Ethan Furman wrote:
>>
>> So I like ExecutionContext for the stack of WhateverWeCallTheOtherContext
>> contexts.  But what do we call it?
>
> How about ExecutionContextFrame, by analogy with stack/stack frame.

My latest suggestion to Yury was to see how the PEP reads with it
called ImplicitContext, such that:

* the active execution context is a stack of implicit contexts
* ContextKey.set() updates the innermost implicit context
* Contextkey.get() reads the whole stack of active implicit contexts
* by default, generators (both sync and async) would have their own
implicit context, but you could make them use the context of method
callers by doing "gen.__implicit_context__ = None"
* by default, coroutines would use their method caller's context, but
async frameworks would make sure to give top-level tasks their own
independent contexts

That proposal came from an initial attempt at redrafting the Abstract
and Rationale sections, where it turns out that one of the things the
current version of the PEP is somewhat taking for granted is that the
reader already has a particular understanding of the difference
between explicit state management (i.e. passing things around as
function arguments and instance attributes) and implicit state
management (i.e. relying on process globals and thread locals).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Koos Zevenhoven]

On Tue, Aug 22, 2017 at 12:44 AM, Yury Selivanov 
wrote:

> On Mon, Aug 21, 2017 at 5:39 PM, Koos Zevenhoven 
> wrote:
> [..]
> >> In the current version of the PEP, generators are initialized with an
> >> empty LogicalContext.  When they are being iterated (started or
> >> resumed), their LogicalContext is pushed to the EC.  When the
> >> iteration is stopped (or paused), they pop their LC from the EC.
> >>
> >
> > Another quick one before I go: Do we really need to push and pop a LC on
> > each next() call, even if it most likely will never be touched?
>
> Yes, otherwise it will be hard to maintain the consistency of the stack.
>
> There will be an optimization: if the LC is empty, we will push NULL
> to the stack, thus avoiding the cost of allocating an object.
>
> ​
But if LCs are immutable, there needs to be only one empty-LC instance.
That would avoid special-casing NULL in code.

​-- Koos​




> I measured the overhead -- generators will become 0.5-1% slower in
> microbenchmarks, but only when they do pretty much nothing. If a
> generator contains more Python code than a bare "yield" expression,
> the overhead will be harder to detect.




-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Yury Selivanov]

On Tue, Aug 22, 2017 at 2:06 AM, Koos Zevenhoven  wrote:
[..]
>> There will be an optimization: if the LC is empty, we will push NULL
>> to the stack, thus avoiding the cost of allocating an object.
>>
> But if LCs are immutable, there needs to be only one empty-LC instance. That
> would avoid special-casing NULL in code.

Yes, this is true.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3

[Guido van Rossum]

On Mon, Aug 21, 2017 at 10:09 PM, Nick Coghlan  wrote:

> My latest suggestion to Yury was to see how the PEP reads with it
> called ImplicitContext, such that:
>
> * the active execution context is a stack of implicit contexts
> * ContextKey.set() updates the innermost implicit context
> * Contextkey.get() reads the whole stack of active implicit contexts
> * by default, generators (both sync and async) would have their own
> implicit context, but you could make them use the context of method
> callers by doing "gen.__implicit_context__ = None"
> * by default, coroutines would use their method caller's context, but
> async frameworks would make sure to give top-level tasks their own
> independent contexts
>
> That proposal came from an initial attempt at redrafting the Abstract
> and Rationale sections, where it turns out that one of the things the
> current version of the PEP is somewhat taking for granted is that the
> reader already has a particular understanding of the difference
> between explicit state management (i.e. passing things around as
> function arguments and instance attributes) and implicit state
> management (i.e. relying on process globals and thread locals).
>

I think I like ImplicitContext. Maybe we can go with this as a working
title at least.

I think we should also rethink the form the key framework-facing APIs will
take, and how they are presented in the PEP -- I am now leaning towards
explaining this from the start as an immutable linked list of immutable
mappings, where the OS-thread state gets updated to a new linked list when
it is changed (either by ContextKey.set or by the various stack
manipulations). I think this falls under several Zen-of-Python points:
EIBTI, and "If the implementation is easy to explain, it may be a good
idea."

-- 
--Guido van Rossum (python.org/~guido )
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Antoine Pitrou]

On Mon, 21 Aug 2017 01:45:05 -0400
"Jim J. Jewett"  wrote:
> Building on Brett's suggestion:
> 
> FrameContext: used in/writable by one frame

It's not frame-specific, it's actually shared by an arbitrary number of
frames (by default, all frames in a given thread).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Ethan Furman]

On 08/21/2017 04:43 AM, Antoine Pitrou wrote:

On Mon, 21 Aug 2017 01:45:05 -0400
"Jim J. Jewett"  wrote:

Building on Brett's suggestion:

 FrameContext: used in/writable by one frame


It's not frame-specific, it's actually shared by an arbitrary number of
frames (by default, all frames in a given thread).


You're thinking too specifically.  A FrameContext/LogicalContext/LocalContext/etc is just a larger frame; although I 
would go with ExecutionContext/ContextFrame, myself.


--
~Ethan~

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Yury Selivanov]

-1 on using "frame" in PEP 550 terminology. Antoine is right, the API
is not frame-specific, and "frame" in Python has only one meaning.

I can certainly see how "ContextFrame" can be correct if we think
about "frame" as a generic term, but in Python, people will
inadvertently think about a connection with frame objects/stacks.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Greg Ewing]

Yury Selivanov wrote:

I can certainly see how "ContextFrame" can be correct if we think
about "frame" as a generic term, but in Python, people will
inadvertently think about a connection with frame objects/stacks.


Calling it ExecutionContextFrame rather than just ContextFrame
would make it clear that it relates to ExecutionContexts in
particular.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Nick Coghlan]

On 22 August 2017 at 10:02, Greg Ewing  wrote:
> Yury Selivanov wrote:
>>
>> I can certainly see how "ContextFrame" can be correct if we think
>> about "frame" as a generic term, but in Python, people will
>> inadvertently think about a connection with frame objects/stacks.
>
> Calling it ExecutionContextFrame rather than just ContextFrame
> would make it clear that it relates to ExecutionContexts in
> particular.

Please, no - it's already hard enough to help people internalise
sync/async design concepts without also introducing ambiguity into the
meaning of terms like locals & frame. Instead, let's leave those as
purely referring to their existing always-synchronous concepts and
find another suitable term for the dynamically nested read/write
mappings making up the ExecutionContext :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Guido van Rossum]

As I understand the key APIs and constraints of the proposal better, I'm
leaning towards FooContext (LC) and FooContextStack (EC), for some value of
Foo that I haven't determined yet. Perhaps the latter can be shortened to
just ContextStack (since the Foo part can probably be guessed from context.
:-)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Greg Ewing]

Guido van Rossum wrote:
Perhaps the latter can be 
shortened to just ContextStack (since the Foo part can probably be 
guessed from context. :-)


-0.9, if I saw something called ContextStack turn up in a traceback
I wouldn't necessarily jump to the conclusion that it was a stack
of FooContexts rather than some other kind of context. EIBTI here,
I think.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[MRAB]

On 2017-08-23 00:12, Greg Ewing wrote:

Guido van Rossum wrote:
Perhaps the latter can be 
shortened to just ContextStack (since the Foo part can probably be 
guessed from context. :-)


-0.9, if I saw something called ContextStack turn up in a traceback
I wouldn't necessarily jump to the conclusion that it was a stack
of FooContexts rather than some other kind of context. EIBTI here,
I think.

The PEP says """This PEP proposes a new mechanism to manage execution 
state""", so ExecutionState or ExecState?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Nathaniel Smith]

On Tue, Aug 22, 2017 at 8:22 AM, Guido van Rossum  wrote:
> As I understand the key APIs and constraints of the proposal better, I'm
> leaning towards FooContext (LC) and FooContextStack (EC), for some value of
> Foo that I haven't determined yet. Perhaps the latter can be shortened to
> just ContextStack (since the Foo part can probably be guessed from context.
> :-)

I guess I'll put in another vote for DynamicContext and
DynamicContextStack. Though my earlier suggestion of these [1] sank
like a stone, either because no-one saw it or because everyone hated
it :-).

We could do worse than just plain Context and ContextStack, for that matter.

-n

[1] https://mail.python.org/pipermail/python-ideas/2017-August/046840.html

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Guido van Rossum]

On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith  wrote:

> We could do worse than just plain Context and ContextStack, for that
> matter.
>

I worry that that's going to lead more people astray thinking this has
something to do with contextlib, which it really doesn't (it's much more
closely related to threading.local()).

Regarding DynamicAnything, I certainly saw it and didn't like it -- the
only place where I've ever seen dynamic scoping was in Emacs Lisp, and I
believe was first shown to me as anti-pattern thirty years ago.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Nathaniel Smith]

On Tue, Aug 22, 2017 at 8:51 PM, Guido van Rossum  wrote:
> On Tue, Aug 22, 2017 at 7:12 PM, Nathaniel Smith  wrote:
>>
>> We could do worse than just plain Context and ContextStack, for that
>> matter.
>
>
> I worry that that's going to lead more people astray thinking this has
> something to do with contextlib, which it really doesn't (it's much more
> closely related to threading.local()).

I guess that could be an argument for LocalContext and
LocalContextStack, to go back full circle...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Nick Coghlan]

On 23 August 2017 at 13:51, Guido van Rossum  wrote:
> Regarding DynamicAnything, I certainly saw it and didn't like it -- the only
> place where I've ever seen dynamic scoping was in Emacs Lisp, and I believe
> was first shown to me as anti-pattern thirty years ago.

As the original proponent of a Dynamic* naming scheme, I'll note that
I eventually decided I didn't like it for the same reason I already
didn't like naming schemes using either the word "local" or the word
"frame": they all suggest a coupling to the synchronous call stack
that deliberately isn't part of the proposal.

That is (at least for me):

- "local" suggests the implicit context will change when locals() changes
- "frame" suggests the implicit context will change when the running
frame changes
- "dynamic" suggest dynamic scoping, which again suggests the implicit
context will change on every function call

None of those associations are true, since the essential point of the
proposal is to *share* implicit state between frames of execution. The
tricky part is defining exactly which frames should and shouldn't
share their implicit context - threading.locals() is our current best
attempt, and the PEP is mainly driven by the problems introduced by
relying solely on that in programs that implement concurrent execution
of Python code independently of the underlying operating system
threads.

My concern with "logical" context is different, which is that as a
term it feels too *broad* to me: I'd expect the lexical context
(nonlocals, module globals), the explicit context (function
parameters), and the runtime context (process globals,
threading.locals()) to also be considered part of the overall logical
context. It's also a very computer-sciencey term of art - if you ask
an arbitrary English speaker "What does 'logical' mean?", they're very
*un*likely to give you an answer that has anything to do with logical
threads of control in computer programs.

My latest suggestion (ImplicitContext) has some of the same issues as
"logical context" (since the runtime context is also implicit), but
seems more defensible on the grounds of it aiming to be a more robust
way of accessing and manipulating implicit state in concurrent
programs, rather than it being the only kind of implicit state that
exists in an application. As an English word, the sense of "capable of
being understood from something else though unexpressed" (the first
definition that Merriam-Webster give) also has the benefit of
describing *exactly* what the PEP is aiming to achieve.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Guido van Rossum]

On Tue, Aug 22, 2017 at 11:21 PM, Nick Coghlan  wrote:
>
> As the original proponent of a Dynamic* naming scheme, I'll note that
> I eventually decided I didn't like it for the same reason I already
> didn't like naming schemes using either the word "local" or the word
> "frame": they all suggest a coupling to the synchronous call stack
> that deliberately isn't part of the proposal.
>

However, there is still *some* connection with the frame, though it's
subtle. In particular the LC (using PEP v3 terms) for a generator is tied
to that generator's frame (it becomes the top of the EC stack whenever that
generator is resumed). I don't think we should call this out in the naming
scheme though.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Antoine Pitrou]

For the record, I'm starting to think that "implicit context" is a
reasonable name.

(in case anyone is interested in those 2 cents of mine :-))

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Yury Selivanov]

I think that "implicit context" is not an accurate description of what
LogicalContext is.

"implicit context" only makes sense when we talk about decimal
context.  For instance, in:

Decimal(1) + Decimal(2)

decimal context is implicit.  But this is "implicit" from the
standpoint of that code.  Decimal will manage its context
*explicitly*.

Fixing decimal context is only one part of the PEP though.  EC will
also allow to implement asynchronous task locals:

current_request = new_context_key()

async def handle_http_request(request):
 current_request.set(request)

Here we explicitly set and will explicitly get values from the EC.  We
will explicitly manage the EC in asyncio.Task and when we schedule
callbacks.

Values stored in the EC are essentially globals (or TLS), which we
don't call "implicit" in Python.

PEP 550 calls generators and asynchronous tasks as "logical threads",
and "logical context" stems directly from that notion.

"implicit" only covers one part of what LC really is. It implies that
all EC/LC operations are implicit, which they are not.

Logical Context is a neutral abstract term.  Implicit Context bears a
specific meaning that I don't think is accurate.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Antoine Pitrou]

On Wed, 23 Aug 2017 12:19:40 -0400
Yury Selivanov  wrote:
> PEP 550 calls generators and asynchronous tasks as "logical threads",
> and "logical context" stems directly from that notion.

I wouldn't refer to a generator as a "thread" personally.  A thread
essentially executes without intervention from outside code, which is
not the case for a generator (where resumption is controlled by
explicit next() calls or iteration by the consumer).

That is also why I proposed the more general "task" (and "task
context").  Of course, capitalized "Task" is given a specific
meaning by asyncio...

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Yury Selivanov]

On Wed, Aug 23, 2017 at 12:56 PM, Antoine Pitrou  wrote:
> On Wed, 23 Aug 2017 12:19:40 -0400
> Yury Selivanov  wrote:
>> PEP 550 calls generators and asynchronous tasks as "logical threads",
>> and "logical context" stems directly from that notion.
>
> I wouldn't refer to a generator as a "thread" personally.  A thread
> essentially executes without intervention from outside code, which is
> not the case for a generator (where resumption is controlled by
> explicit next() calls or iteration by the consumer).

I agree. That's why we are rewriting the PEP without mentioning
"logical threads".

>
> That is also why I proposed the more general "task" (and "task
> context").  Of course, capitalized "Task" is given a specific
> meaning by asyncio...

Yeah.. I like TaskContext when it's applied to asynchronous code. It
doesn't really work for generators because we never refer to
generators as tasks.

Out of what was proposed so far to replace Logical Context:

1. DynamicContext: close to "dynamic scoping", but tries to avoid
mentioning "scopes". There are only a few languages where dynamic
scoping is implemented, so people are generally not aware of it.

2. ContextFrame and all frame-related names: implies that EC is
somehow implemented on top of frames or is frame-specific (which is
not always true).

3. ImplicitContext: covers one specific property observed from
specific examples.  Context in PEP 550 is managed explicitly in some
cases. There are many use cases when API users will be working with it
explicitly too (to wrirte/read data from it). FWIW I believe that
"ExplicitContext" would be more accurate than "ImplicitContext".

4. LocalContext: we use "local" to describe local variables and
scoping in Python, we want to avoid any confusion here.

5. TaskContext: works for asynchronous tasks, but not for generators.

I don't think that replacing LogicalContext with any name in this list
will make any improvement.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Guido van Rossum]

In favor of ImplicitContext is one point: it is indeed "implicit" if you
compare it with the "explicit" way of passing state around, which would
require an extra argument containing the state for any function that uses
the state *or calls a function that uses the state* (recursively).

On Wed, Aug 23, 2017 at 11:27 AM, Yury Selivanov 
wrote:

> On Wed, Aug 23, 2017 at 12:56 PM, Antoine Pitrou 
> wrote:
> > On Wed, 23 Aug 2017 12:19:40 -0400
> > Yury Selivanov  wrote:
> >> PEP 550 calls generators and asynchronous tasks as "logical threads",
> >> and "logical context" stems directly from that notion.
> >
> > I wouldn't refer to a generator as a "thread" personally.  A thread
> > essentially executes without intervention from outside code, which is
> > not the case for a generator (where resumption is controlled by
> > explicit next() calls or iteration by the consumer).
>
> I agree. That's why we are rewriting the PEP without mentioning
> "logical threads".
>
> >
> > That is also why I proposed the more general "task" (and "task
> > context").  Of course, capitalized "Task" is given a specific
> > meaning by asyncio...
>
> Yeah.. I like TaskContext when it's applied to asynchronous code. It
> doesn't really work for generators because we never refer to
> generators as tasks.
>
> Out of what was proposed so far to replace Logical Context:
>
> 1. DynamicContext: close to "dynamic scoping", but tries to avoid
> mentioning "scopes". There are only a few languages where dynamic
> scoping is implemented, so people are generally not aware of it.
>
> 2. ContextFrame and all frame-related names: implies that EC is
> somehow implemented on top of frames or is frame-specific (which is
> not always true).
>
> 3. ImplicitContext: covers one specific property observed from
> specific examples.  Context in PEP 550 is managed explicitly in some
> cases. There are many use cases when API users will be working with it
> explicitly too (to wrirte/read data from it). FWIW I believe that
> "ExplicitContext" would be more accurate than "ImplicitContext".
>
> 4. LocalContext: we use "local" to describe local variables and
> scoping in Python, we want to avoid any confusion here.
>
> 5. TaskContext: works for asynchronous tasks, but not for generators.
>
> I don't think that replacing LogicalContext with any name in this list
> will make any improvement.
>
> Yury
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Ethan Furman]

On 08/23/2017 11:27 AM, Yury Selivanov wrote:


Out of what was proposed so far to replace Logical Context:

[...]

I don't think that replacing LogicalContext with any name in this list
will make any improvement.


How about ExecutionContext and ContextVars ?

We are already used to different levels of variables: global, local, non-local, class.  I think having variables tied to 
a Context, and having search flow back to previous Contexts, would be easy to understand.


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Yury Selivanov]

On Wed, Aug 23, 2017 at 2:47 PM, Guido van Rossum  wrote:
> In favor of ImplicitContext is one point: it is indeed "implicit" if you
> compare it with the "explicit" way of passing state around, which would
> require an extra argument containing the state for any function that uses
> the state *or calls a function that uses the state* (recursively).

This is a good point, now I understand the reasoning better.

I guess I'm looking at this from a perspective of PEP 550 users.  They
will use either ContextKey-like API described in the PEP now, or
something similar to threading.local() to store values in the context
of a generator or an async task.  From the standpoint of the user,
they work with some sort of of a "global" variable.  To me, renaming
LC to ImplicitContext sounds similar to saying that using global
variables is implicit, or calling "global" scope "implicit".

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Yury Selivanov]

> How about ExecutionContext and ContextVars ?

> We are already used to different levels of variables: global, local, 
> non-local, class.  I think having variables tied to a Context, and having 
> search flow back to previous Contexts, would be easy to understand.

Yeah, I had this idea about ContextVars and so far I like it more than
Context Keys and Context Values.

In the next version of the PEP (will post it soon), we use Execution
Context, Logical Context, and Context Variable terminology (the debate
if we should use other names is, of course, still open).

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Simon Cross]

What about "CoroutineScope" or "CoroutineContext"?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Ethan Furman]

On 08/23/2017 12:17 PM, Yury Selivanov wrote:

How about ExecutionContext and ContextVars ?



We are already used to different levels of variables: global, local, non-local, 
class.  I think having variables tied to a Context, and having search flow back 
to previous Contexts, would be easy to understand.


Yeah, I had this idea about ContextVars and so far I like it more than
Context Keys and Context Values.

In the next version of the PEP (will post it soon), we use Execution
Context, Logical Context, and Context Variable terminology (the debate
if we should use other names is, of course, still open).


ContextVars is actually a different name for LogicalContext.  So it would be:

  ExecutionContext = [ContextVars()[, ContextVars()[ ...]]]

and you get the (thread.local similar) ContextVars by

  context_vars = sys.get_context_vars()   # or whatever
  context_vars.verbose = ...

--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Antoine Pitrou]

On Wed, 23 Aug 2017 14:27:55 -0400
Yury Selivanov  wrote:
> 
> Yeah.. I like TaskContext when it's applied to asynchronous code. It
> doesn't really work for generators because we never refer to
> generators as tasks.
> 
> Out of what was proposed so far to replace Logical Context:
> 
> 1. DynamicContext: close to "dynamic scoping", but tries to avoid
> mentioning "scopes". There are only a few languages where dynamic
> scoping is implemented, so people are generally not aware of it.
> 
> 2. ContextFrame and all frame-related names: implies that EC is
> somehow implemented on top of frames or is frame-specific (which is
> not always true).
> 
> 3. ImplicitContext: covers one specific property observed from
> specific examples.  Context in PEP 550 is managed explicitly in some
> cases. There are many use cases when API users will be working with it
> explicitly too (to wrirte/read data from it). FWIW I believe that
> "ExplicitContext" would be more accurate than "ImplicitContext".
> 
> 4. LocalContext: we use "local" to describe local variables and
> scoping in Python, we want to avoid any confusion here.
> 
> 5. TaskContext: works for asynchronous tasks, but not for generators.
> 
> I don't think that replacing LogicalContext with any name in this list
> will make any improvement.

I think you have a point.  Though every time I want to type "logical
context" it seems my fingers slip and type "local context" instead :-)

Now remains the question of why the logical context stack is named
"execution context" and not "logical context stack" (or "logical
context chain" to keep the ChainMap analogy) :-)

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Nick Coghlan]

On 24 August 2017 at 02:19, Yury Selivanov  wrote:
> I think that "implicit context" is not an accurate description of what
> LogicalContext is.
>
> "implicit context" only makes sense when we talk about decimal
> context.  For instance, in:
>
> Decimal(1) + Decimal(2)
>
> decimal context is implicit.  But this is "implicit" from the
> standpoint of that code.  Decimal will manage its context
> *explicitly*.
>
> Fixing decimal context is only one part of the PEP though.  EC will
> also allow to implement asynchronous task locals:
>
> current_request = new_context_key()
>
> async def handle_http_request(request):
>  current_request.set(request)
>
> Here we explicitly set and will explicitly get values from the EC.  We
> will explicitly manage the EC in asyncio.Task and when we schedule
> callbacks.

The implicit behaviour that "implicit context" refers to is the fact
that if you look at an isolated snippet of code, you have no idea what
context you're actually going to be querying or modifying - that's
implicit in the execution of the whole program.

As a user of the read/write API though, you shouldn't need to care all
that much - the whole point of PEP 550 is to define default behaviours
and guidelines such that it will be something sensible.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Nathaniel Smith]

On Thu, Aug 24, 2017 at 1:22 AM, Nick Coghlan  wrote:
> On 24 August 2017 at 02:19, Yury Selivanov  wrote:
>> I think that "implicit context" is not an accurate description of what
>> LogicalContext is.
>>
>> "implicit context" only makes sense when we talk about decimal
>> context.  For instance, in:
>>
>> Decimal(1) + Decimal(2)
>>
>> decimal context is implicit.  But this is "implicit" from the
>> standpoint of that code.  Decimal will manage its context
>> *explicitly*.
>>
>> Fixing decimal context is only one part of the PEP though.  EC will
>> also allow to implement asynchronous task locals:
>>
>> current_request = new_context_key()
>>
>> async def handle_http_request(request):
>>  current_request.set(request)
>>
>> Here we explicitly set and will explicitly get values from the EC.  We
>> will explicitly manage the EC in asyncio.Task and when we schedule
>> callbacks.
>
> The implicit behaviour that "implicit context" refers to is the fact
> that if you look at an isolated snippet of code, you have no idea what
> context you're actually going to be querying or modifying - that's
> implicit in the execution of the whole program.

How about: RuntimeContext and RuntimeContextStack

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Nick Coghlan]

On 24 August 2017 at 08:47, Ethan Furman  wrote:
>
> ContextVars is actually a different name for LogicalContext.  So it would
> be:
>
>   ExecutionContext = [ContextVars()[, ContextVars()[ ...]]]
>
> and you get the (thread.local similar) ContextVars by
>
>   context_vars = sys.get_context_vars()   # or whatever
>   context_vars.verbose = ...

Migrating a (variant of a) naming subthread from python-ideas over to
here, does the following sound plausible to anyone else?:

ContextLocal - read/write access API (via get()/set() methods)
ContextLocalNamespace - active mapping that CL.get()/set() manipulates
ExecutionContext - current stack of context local namespaces

Building a threading.local() style helper API would then look something like:

class ContextLocals:
def __init__(self, key_prefix):
self._key_prefix = key_prefix

def __getattr__(self, attr):
debugging_name = "{}.{}".format(self._key_prefix, attr)
self.__dict__[attr] = new_local =
sys.new_context_local(debugging_name)
return new_local

def __setattr__(self, attr, value):
getattr(self, attr).set(value)

def __delattr__(self, attr):
getattr(self, attr).set(None)

my_state = ContextLocals(__name__)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Guido van Rossum]

It shouldn't be called a namespace unless the dominant access is via
attributes.

On Thu, Aug 24, 2017 at 4:37 AM, Nick Coghlan  wrote:

> On 24 August 2017 at 08:47, Ethan Furman  wrote:
> >
> > ContextVars is actually a different name for LogicalContext.  So it would
> > be:
> >
> >   ExecutionContext = [ContextVars()[, ContextVars()[ ...]]]
> >
> > and you get the (thread.local similar) ContextVars by
> >
> >   context_vars = sys.get_context_vars()   # or whatever
> >   context_vars.verbose = ...
>
> Migrating a (variant of a) naming subthread from python-ideas over to
> here, does the following sound plausible to anyone else?:
>
> ContextLocal - read/write access API (via get()/set() methods)
> ContextLocalNamespace - active mapping that CL.get()/set() manipulates
> ExecutionContext - current stack of context local namespaces
>
> Building a threading.local() style helper API would then look something
> like:
>
> class ContextLocals:
> def __init__(self, key_prefix):
> self._key_prefix = key_prefix
>
> def __getattr__(self, attr):
> debugging_name = "{}.{}".format(self._key_prefix, attr)
> self.__dict__[attr] = new_local =
> sys.new_context_local(debugging_name)
> return new_local
>
> def __setattr__(self, attr, value):
> getattr(self, attr).set(value)
>
> def __delattr__(self, attr):
> getattr(self, attr).set(None)
>
> my_state = ContextLocals(__name__)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Eric Snow]

On Thu, Aug 24, 2017 at 5:37 AM, Nick Coghlan  wrote:
> Migrating a (variant of a) naming subthread from python-ideas over to
> here, does the following sound plausible to anyone else?:
>
> ContextLocal - read/write access API (via get()/set() methods)
> ContextLocalNamespace - active mapping that CL.get()/set() manipulates
> ExecutionContext - current stack of context local namespaces

Overall it makes sense.  The following derivative might be more clear:

ContextLocal
ContextLocals
ChainedContextLocals (or some variation explicitly calling out the chaining)

Alternatively, similar to other suggestions:

ContextVar
ContextVars
ChainedContextVars

I also think "state" helps disambiguate "context":

ContextVar
ContextState
ChainedContextState

or:

ContextVar
ContextVars
ContextState (perhaps a little too ambiguous without the explicit "chained")

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Nick Coghlan]

On 25 August 2017 at 01:00, Guido van Rossum  wrote:
> It shouldn't be called a namespace unless the dominant access is via
> attributes.

That makes sense.

Since the main purpose of that part of the Python API is to offer an
opaque handle to where the context locals store their values,
something semantically neutral like "State" may work:

- ContextLocal: read/write API
- ContextLocalState: where ContextLocal instances actually store things
- ExecutionContext: nested stack of context local states

The attribute on generators and coroutines could then be called
"__context_locals__", and that would either be None (indicating that
any context local references will just use the already active context
local storage), or else it would be set to a ContextLocalState
instance (indicate that starting & stopping the operation will push &
pop the given context local state).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 550 v3 naming

[Guido van Rossum]

I think we should let the naming issue go for now, while Yury et al. are
rewriting the PEP. We'll revisit it after we're more comfortable with the
semantics.

On Thu, Aug 24, 2017 at 9:55 PM, Nick Coghlan  wrote:

> On 25 August 2017 at 01:00, Guido van Rossum  wrote:
> > It shouldn't be called a namespace unless the dominant access is via
> > attributes.
>
> That makes sense.
>
> Since the main purpose of that part of the Python API is to offer an
> opaque handle to where the context locals store their values,
> something semantically neutral like "State" may work:
>
> - ContextLocal: read/write API
> - ContextLocalState: where ContextLocal instances actually store things
> - ExecutionContext: nested stack of context local states
>
> The attribute on generators and coroutines could then be called
> "__context_locals__", and that would either be None (indicating that
> any context local references will just use the already active context
> local storage), or else it would be set to a ContextLocalState
> instance (indicate that starting & stopping the operation will push &
> pop the given context local state).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com