Re: [Python-Dev] PEP 550 v3 naming
On Mon, 21 Aug 2017 01:45:05 -0400 "Jim J. Jewett" wrote: > Building on Brett's suggestion: > > FrameContext: used in/writable by one frame It's not frame-specific, it's actually shared by an arbitrary number of frames (by default, all frames in a given thread). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On 21 August 2017 at 15:03, Guido van Rossum wrote:
> Honestly I'm not sure we need the distinction between LC and EC. If you read
> carefully some of the given example code seems to confuse them. If we could
> get away with only a single framework-facing concept, I would be happy
> calling it ExecutionContext.
Unfortunately, I don't think we can, and that's why I tried to reframe
the discussion in terms of "Where ContextKey.set() writes to" and
"Where ContextKey.get() looks things up".
Consider the following toy generator:
def tracking_gen():
start_tracking_iterations()
while True:
tally_iteration()
yield
task_id = ContextKey("task_id")
iter_counter = ContextKey("iter_counter")
def start_tracking_iterations():
iter_counter.set(collection.Counter())
def tally_iteration():
current_task = task_id.get() # Set elsewhere
iter_counter.get()[current_task] += 1
Now, this isn't a very *sensible* generator (since it could just use a
regular object instance for tracking instead of a context variable),
but nevertheless, it's one that we would expect to work, and it's one
that we would expect to exhibit the following properties:
1. When tally_iteration() calls task_id.get(), we expect that to be
resolved in the context calling next() on the instance, *not* the
context where the generator was first created
2. When tally_iteration() calls iter_counter.get(), we expect that to
be resolved in the same context where start_tracking_iterations()
called iter_counter.set()
This has consequences for the design in the PEP:
* what we want to capture at generator creation time is the context
where writes will happen, and we also want that to be the innermost
context used for lookups
* other than that innermost context, we want everything else to be dynamic
* this means that "mutable context saved on the generator" and "entire
dynamic context visible when the generator runs" aren't the same thing
And hence the introduction of the LocalContext/LogicalContext
terminology for the former, and the ExecutionContext terminology for
the latter.
It's also where the analogy with ChainMap came from (although I don't
think this has made it into the PEP itself):
* LogicalContext is the equivalent of the individual mappings
* ExecutionContext is the equivalent of ChainMap
* ContextKey.get() replaces ChainMap.__getitem__
* ContextKey.set(value) replaces ChainMap.__setitem__
* ContextKey.set(None) replaces ChainMap.__delitem__
While the context is defined conceptually as a nested chain of
key:value mappings, we avoid using the mapping syntax because of the
way the values can shift dynamically out from under you based on who
called you - while the ChainMap analogy is hopefully helpful to
understanding, we don't want people taking it too literally or things
will become more confusing rather than less.
Despite that risk, taking the analogy further is where the
DynamicWriteContext + DynamicLookupContext terminology idea came from:
* like ChainMap.new_child(), adjusting the DynamicWriteContext changes
what ck.set() affects, and also sets the innermost context for
ck.get()
* like using a different ChainMap, adjusting the DynamicLookupContext
changes what ck.get() can see (unlike ChainMap, it also isolates
ck.set() by default)
I'll also note that the first iteration of the PEP didn't really make
this distinction, and it caused a problem that Nathaniel pointed out:
generators would "snapshot" their entire dynamic context when first
created, and then never adjust it for external changes between
iterations. This meant that if you adjusted something like the decimal
context outside the generator after creating it, it would ignore those
changes - instead of having the problem of changes inside the
generator leaking out, we instead had the problem of changes outside
the generator *not* making their way in, even if you wanted them to.
Due to that heritage, fixing some of the examples could easily have
been missed in the v2 rewrite that introduced the distinction between
the two kinds of context.
> (Another critique of the proposal I have is that it adds too many
> similarly-named functions to sys. But this email is already too long and I
> need to go to bed.)
If it helps any, one of the ideas that has come up is to put all of
the proposed context manipulation APIs in contextlib rather than in
sys, and I think that's a reasonable idea (I don't think any of us
actually like the notion of adding that many new subsystem specific
APIs directly to sys).
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
On 08/21/2017 04:43 AM, Antoine Pitrou wrote: On Mon, 21 Aug 2017 01:45:05 -0400 "Jim J. Jewett" wrote: Building on Brett's suggestion: FrameContext: used in/writable by one frame It's not frame-specific, it's actually shared by an arbitrary number of frames (by default, all frames in a given thread). You're thinking too specifically. A FrameContext/LogicalContext/LocalContext/etc is just a larger frame; although I would go with ExecutionContext/ContextFrame, myself. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
-1 on using "frame" in PEP 550 terminology. Antoine is right, the API is not frame-specific, and "frame" in Python has only one meaning. I can certainly see how "ContextFrame" can be correct if we think about "frame" as a generic term, but in Python, people will inadvertently think about a connection with frame objects/stacks. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Mon, Aug 21, 2017 at 7:12 AM, Nick Coghlan wrote:
> On 21 August 2017 at 15:03, Guido van Rossum wrote:
> > Honestly I'm not sure we need the distinction between LC and EC. If you
> read
> > carefully some of the given example code seems to confuse them. If we
> could
> > get away with only a single framework-facing concept, I would be happy
> > calling it ExecutionContext.
>
> Unfortunately, I don't think we can, and that's why I tried to reframe
> the discussion in terms of "Where ContextKey.set() writes to" and
> "Where ContextKey.get() looks things up".
>
> Consider the following toy generator:
>
> def tracking_gen():
> start_tracking_iterations()
> while True:
> tally_iteration()
> yield
>
> task_id = ContextKey("task_id")
> iter_counter = ContextKey("iter_counter")
>
> def start_tracking_iterations():
> iter_counter.set(collection.Counter())
>
> def tally_iteration():
> current_task = task_id.get() # Set elsewhere
> iter_counter.get()[current_task] += 1
>
> Now, this isn't a very *sensible* generator (since it could just use a
> regular object instance for tracking instead of a context variable),
> but nevertheless, it's one that we would expect to work, and it's one
> that we would expect to exhibit the following properties:
>
> 1. When tally_iteration() calls task_id.get(), we expect that to be
> resolved in the context calling next() on the instance, *not* the
> context where the generator was first created
> 2. When tally_iteration() calls iter_counter.get(), we expect that to
> be resolved in the same context where start_tracking_iterations()
> called iter_counter.set()
>
> This has consequences for the design in the PEP:
>
> * what we want to capture at generator creation time is the context
> where writes will happen, and we also want that to be the innermost
> context used for lookups
> * other than that innermost context, we want everything else to be dynamic
> * this means that "mutable context saved on the generator" and "entire
> dynamic context visible when the generator runs" aren't the same thing
>
> And hence the introduction of the LocalContext/LogicalContext
> terminology for the former, and the ExecutionContext terminology for
> the latter.
>
OK, this is a sensible explanation. I think the PEP would benefit from
including some version of it early on (though perhaps shortened a bit).
> It's also where the analogy with ChainMap came from (although I don't
> think this has made it into the PEP itself):
>
> * LogicalContext is the equivalent of the individual mappings
> * ExecutionContext is the equivalent of ChainMap
> * ContextKey.get() replaces ChainMap.__getitem__
> * ContextKey.set(value) replaces ChainMap.__setitem__
> * ContextKey.set(None) replaces ChainMap.__delitem__
>
> While the context is defined conceptually as a nested chain of
> key:value mappings, we avoid using the mapping syntax because of the
> way the values can shift dynamically out from under you based on who
> called you - while the ChainMap analogy is hopefully helpful to
> understanding, we don't want people taking it too literally or things
> will become more confusing rather than less.
>
Agreed. However now I am confused as to how the HAMT fits in. Yury says
somewhere that the HAMT will be used for the EC and then cloning the EC is
just returning a pointer to the same EC. But even if I interpret that as
making a new EC containing a pointer to the same underlying HAMT, I don't
see how that will preserve the semantics that different logical threads,
running interleaved (like different generators being pumped alternatingly),
will see updates to LCs that are lower on the stack of LCs in the EC. (I
see this with the stack-of-dicts version, but not with the immutable HAMT
inplementation.)
> Despite that risk, taking the analogy further is where the
> DynamicWriteContext + DynamicLookupContext terminology idea came from:
>
> * like ChainMap.new_child(), adjusting the DynamicWriteContext changes
> what ck.set() affects, and also sets the innermost context for
> ck.get()
> * like using a different ChainMap, adjusting the DynamicLookupContext
> changes what ck.get() can see (unlike ChainMap, it also isolates
> ck.set() by default)
>
Here I'm lost again. In the PEP's pseudo code, your first bullet seems to
be the operation "push a new LC on the stack of the current EC". Does the
second bullet just mean "switch to a different EC"?
> I'll also note that the first iteration of the PEP didn't really make
> this distinction, and it caused a problem that Nathaniel pointed out:
> generators would "snapshot" their entire dynamic context when first
> created, and then never adjust it for external changes between
> iterations. This meant that if you adjusted something like the decimal
> context outside the generator after creating it, it would ignore those
> changes - instead of having the problem of changes inside the
> generator
Re: [Python-Dev] PEP 550 v3
On Mon, Aug 21, 2017 at 3:10 PM, Guido van Rossum wrote:
[..]
> Agreed. However now I am confused as to how the HAMT fits in. Yury says
> somewhere that the HAMT will be used for the EC and then cloning the EC is
> just returning a pointer to the same EC. But even if I interpret that as
> making a new EC containing a pointer to the same underlying HAMT, I don't
> see how that will preserve the semantics that different logical threads,
> running interleaved (like different generators being pumped alternatingly),
> will see updates to LCs that are lower on the stack of LCs in the EC. (I see
> this with the stack-of-dicts version, but not with the immutable HAMT
> inplementation.)
Few important things (using the current PEP 550 terminology):
* ExecutionContext is a *dynamic* stack of LogicalContexts.
* LCs do not reference other LCs.
* ContextKey.set() can only modify the *top* LC in the stack.
If LC is a mutable mapping:
# EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
a.set(c)
#LC4 = EC.top()
#LC4[a] = c
# EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})]
If LC are implemented with immutable mappings:
# EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
a.set(c)
#LC4 = EC.pop()
#LC4_1 = LC4.copy()
#LC4_1[a] = c
#EC.push(LC4_1)
# EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})]
Any code that uses EC will not see any difference, because it can only
work with the top LC.
Back to generators. Generators have their own empty LCs when created
to store their *local* EC modifications.
When a generator is *being* iterated, it pushes its LC to the EC. When
the iteration step is finished, it pops its LC from the EC. If you
have nested generators, they will dynamically build a stack of their
LCs while they are iterated.
Therefore, generators *naturally* control the stack of EC. We can't
execute two generators simultaneously in one thread (we can only
iterate them one by one), so the top LC always belongs to the current
generator that is being iterated:
def nested_gen():
# EC = [outer_LC, gen1_LC, nested_gen_LC]
yield
# EC = [outer_LC, gen1_LC, nested_gen_LC]
yield
def gen1():
# EC = [outer_LC, gen1_LC]
n = nested_gen()
yield
# EC = [outer_LC, gen1_LC]
next(n)
# EC = [outer_LC, gen1_LC]
yield
next(n)
# EC = [outer_LC, gen1_LC]
def gen2():
# EC = [outer_LC, gen2_LC]
yield
# EC = [outer_LC, gen2_LC]
yield
g1 = gen1()
g2 = gen2()
next(g1)
next(g2)
next(g1)
next(g2)
HAMT is a way to efficiently implement immutable mappings with O(log32
N) set operation, that's it. If we implement immutable mappings using
regular dicts and copy, set() would be O(log N).
[..]
>>
>> I'll also note that the first iteration of the PEP didn't really make
>> this distinction, and it caused a problem that Nathaniel pointed out:
>> generators would "snapshot" their entire dynamic context when first
>> created, and then never adjust it for external changes between
>> iterations. This meant that if you adjusted something like the decimal
>> context outside the generator after creating it, it would ignore those
>> changes - instead of having the problem of changes inside the
>> generator leaking out, we instead had the problem of changes outside
>> the generator *not* making their way in, even if you wanted them to.
>
>
> OK, this really needs to be made very clear early in the PEP. Maybe this
> final sentence provides the key requirement: changes outside the generator
> should make it into the generator when next() is invoked, unless the
> generator itself has made an override; but changes inside the generator
> should not leak out through next().
It's covered here with two examples:
https://www.python.org/dev/peps/pep-0550/#ec-semantics-for-generators
Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Sat, Aug 19, 2017 at 4:17 AM, Nick Coghlan wrote: [..] >> >> * Generator's ``.send()`` and ``.throw()`` methods are modified as >> follows (in pseudo-C):: >> >> if gen.__logical_context__ is not NULL: >> tstate = PyThreadState_Get() >> >> tstate.execution_context.push(gen.__logical_context__) >> >> try: >> # Perform the actual `Generator.send()` or >> # `Generator.throw()` call. >> return gen.send(...) >> finally: >> gen.__logical_context__ = tstate.execution_context.pop() >> else: >> # Perform the actual `Generator.send()` or >> # `Generator.throw()` call. >> return gen.send(...) > > I think this pseudo-code expansion includes a few holdovers from the > original visibly-immutable API design. > > Given the changes since then, I think this would be clearer if the > first branch used sys.run_with_logical_context(), since the logical > context references at the Python layer now behave like shared mutable > objects, and the apparent immutability of > sys.run_with_execution_context() comes from injecting a fresh logical > context every time. This is a good idea, I like it! It will indeed simplify the explanation. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Mon, Aug 21, 2017 at 5:12 PM, Nick Coghlan wrote:
> On 21 August 2017 at 15:03, Guido van Rossum wrote:
> > Honestly I'm not sure we need the distinction between LC and EC. If you
> read
> > carefully some of the given example code seems to confuse them. If we
> could
> > get away with only a single framework-facing concept, I would be happy
> > calling it ExecutionContext.
>
> Unfortunately, I don't think we can, and that's why I tried to reframe
> the discussion in terms of "Where ContextKey.set() writes to" and
> "Where ContextKey.get() looks things up".
>
> Consider the following toy generator:
>
> def tracking_gen():
> start_tracking_iterations()
> while True:
> tally_iteration()
> yield
>
> task_id = ContextKey("task_id")
> iter_counter = ContextKey("iter_counter")
>
> def start_tracking_iterations():
> iter_counter.set(collection.Counter())
>
> def tally_iteration():
> current_task = task_id.get() # Set elsewhere
> iter_counter.get()[current_task] += 1
>
> Now, this isn't a very *sensible* generator (since it could just use a
> regular object instance for tracking instead of a context variable),
> but nevertheless, it's one that we would expect to work, and it's one
> that we would expect to exhibit the following properties:
>
> 1. When tally_iteration() calls task_id.get(), we expect that to be
> resolved in the context calling next() on the instance, *not* the
> context where the generator was first created
> 2. When tally_iteration() calls iter_counter.get(), we expect that to
> be resolved in the same context where start_tracking_iterations()
> called iter_counter.set()
>
> This has consequences for the design in the PEP:
>
> * what we want to capture at generator creation time is the context
> where writes will happen, and we also want that to be the innermost
> context used for lookups
>
I don't get it. How is this a consequence of the above two points? And why
do we need to capture something (a "context") at generator creation time?
-- Koos
> * other than that innermost context, we want everything else to be dynamic
> * this means that "mutable context saved on the generator" and "entire
> dynamic context visible when the generator runs" aren't the same thing
>
> And hence the introduction of the LocalContext/LogicalContext
> terminology for the former, and the ExecutionContext terminology for
> the latter.
>
>
> [...]
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Mon, Aug 21, 2017 at 5:14 PM, Koos Zevenhoven wrote: [..] >> This has consequences for the design in the PEP: >> >> * what we want to capture at generator creation time is the context >> where writes will happen, and we also want that to be the innermost >> context used for lookups > > > I don't get it. How is this a consequence of the above two points? And why > do we need to capture something (a "context") at generator creation time? > We don't need to "capture" anything when a generator is created (it was something that PEP 550 version 1 was doing). In the current version of the PEP, generators are initialized with an empty LogicalContext. When they are being iterated (started or resumed), their LogicalContext is pushed to the EC. When the iteration is stopped (or paused), they pop their LC from the EC. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Tue, Aug 22, 2017 at 12:25 AM, Yury Selivanov wrote: > On Mon, Aug 21, 2017 at 5:14 PM, Koos Zevenhoven > wrote: > [..] > >> This has consequences for the design in the PEP: > >> > >> * what we want to capture at generator creation time is the context > >> where writes will happen, and we also want that to be the innermost > >> context used for lookups > > > > > > I don't get it. How is this a consequence of the above two points? And > why > > do we need to capture something (a "context") at generator creation time? > > > > We don't need to "capture" anything when a generator is created (it > was something that PEP 550 version 1 was doing). > > Ok, good. > In the current version of the PEP, generators are initialized with an > empty LogicalContext. When they are being iterated (started or > resumed), their LogicalContext is pushed to the EC. When the > iteration is stopped (or paused), they pop their LC from the EC. > > Another quick one before I go: Do we really need to push and pop a LC on each next() call, even if it most likely will never be touched? -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Mon, Aug 21, 2017 at 5:39 PM, Koos Zevenhoven wrote: [..] >> In the current version of the PEP, generators are initialized with an >> empty LogicalContext. When they are being iterated (started or >> resumed), their LogicalContext is pushed to the EC. When the >> iteration is stopped (or paused), they pop their LC from the EC. >> > > Another quick one before I go: Do we really need to push and pop a LC on > each next() call, even if it most likely will never be touched? Yes, otherwise it will be hard to maintain the consistency of the stack. There will be an optimization: if the LC is empty, we will push NULL to the stack, thus avoiding the cost of allocating an object. I measured the overhead -- generators will become 0.5-1% slower in microbenchmarks, but only when they do pretty much nothing. If a generator contains more Python code than a bare "yield" expression, the overhead will be harder to detect. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
Yury Selivanov wrote: I can certainly see how "ContextFrame" can be correct if we think about "frame" as a generic term, but in Python, people will inadvertently think about a connection with frame objects/stacks. Calling it ExecutionContextFrame rather than just ContextFrame would make it clear that it relates to ExecutionContexts in particular. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Mon, Aug 21, 2017 at 12:50 PM, Yury Selivanov
wrote:
> Few important things (using the current PEP 550 terminology):
>
> * ExecutionContext is a *dynamic* stack of LogicalContexts.
> * LCs do not reference other LCs.
> * ContextKey.set() can only modify the *top* LC in the stack.
>
> If LC is a mutable mapping:
>
> # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
>
> a.set(c)
> #LC4 = EC.top()
> #LC4[a] = c
>
> # EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})]
>
> If LC are implemented with immutable mappings:
>
> # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
>
> a.set(c)
> #LC4 = EC.pop()
> #LC4_1 = LC4.copy()
> #LC4_1[a] = c
> #EC.push(LC4_1)
>
> # EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})]
>
> Any code that uses EC will not see any difference, because it can only
> work with the top LC.
>
OK, good. This makes more sense, especially if I read "the EC" as shorthand
for the EC stored in the current thread's per-thread state. The immutable
mapping (if used) is used for the LC, not for the EC, and in this case
cloning an EC would simply make a shallow copy of its underlying list --
whereas without the immutable mapping, cloning the EC would also require
making shallow copies of each LC. And I guess the linked-list
implementation (Approach #3 in the PEP) makes EC cloning an O(1) operation.
Note that there is a lot of hand-waving and shorthand in this explanation,
but I think I finally follow the design. It is going to be a big task to
write this up in a didactic way -- the current PEP needs a fair amount of
help in that sense. (If you want to become a better writer, I've recently
enjoyed reading Steven Pinker's *The Sense of Style*: The Thinking Person's
Guide to Writing in the 21st Century. Amongst other fascinating topics, it
explains why so often what we think is clearly written can cause so much
confusion.)
> Back to generators. Generators have their own empty LCs when created
> to store their *local* EC modifications.
>
> When a generator is *being* iterated, it pushes its LC to the EC. When
> the iteration step is finished, it pops its LC from the EC. If you
> have nested generators, they will dynamically build a stack of their
> LCs while they are iterated.
>
> Therefore, generators *naturally* control the stack of EC. We can't
> execute two generators simultaneously in one thread (we can only
> iterate them one by one), so the top LC always belongs to the current
> generator that is being iterated:
>
> def nested_gen():
> # EC = [outer_LC, gen1_LC, nested_gen_LC]
> yield
> # EC = [outer_LC, gen1_LC, nested_gen_LC]
> yield
>
> def gen1():
> # EC = [outer_LC, gen1_LC]
> n = nested_gen()
> yield
> # EC = [outer_LC, gen1_LC]
> next(n)
> # EC = [outer_LC, gen1_LC]
> yield
> next(n)
> # EC = [outer_LC, gen1_LC]
>
> def gen2():
> # EC = [outer_LC, gen2_LC]
> yield
> # EC = [outer_LC, gen2_LC]
> yield
>
> g1 = gen1()
> g2 = gen2()
>
> next(g1)
> next(g2)
> next(g1)
> next(g2)
>
This, combined with your later clarification:
> In the current version of the PEP, generators are initialized with an
> empty LogicalContext. When they are being iterated (started or
> resumed), their LogicalContext is pushed to the EC. When the
> iteration is stopped (or paused), they pop their LC from the EC.
makes it clear how the proposal works for generators. There's an important
piece that I hadn't figured out from Nick's generator example, because I
had mistakenly assumed that something *would* be captured at generator
create time. It's a reasonable mistake to make, I think -- the design space
here is just huge and there are many variations that don't affect typical
code but do differ in edge cases. Your clear statement "nothing needs to be
captured" is helpful to avoid this misunderstanding.
> HAMT is a way to efficiently implement immutable mappings with O(log32
> N) set operation, that's it. If we implement immutable mappings using
> regular dicts and copy, set() would be O(log N).
>
This sounds like abuse of the O() notation. Mathematically O(log N) and
O(log32 N) surely must be equivalent, since log32 N is just K*(log N) for
some constant K (about 0.288539), and the constant disappears in the O(),
as O(K*f(N)) and O(f(N)) are equivalent. Now, I'm happy to hear that a
HAMT-based implementation is faster than a dict+copy-based implementation,
but I don't think your use of O() makes sense here.
> [..]
> >>
> >> I'll also note that the first iteration of the PEP didn't really make
> >> this distinction, and it caused a problem that Nathaniel pointed out:
> >> generators would "snapshot" their entire dynamic context when first
> >> created, and then never adjust it for external changes between
> >> iterations. This meant that if you adjusted something like the
Re: [Python-Dev] PEP 550 v3
On Mon, Aug 21, 2017 at 8:06 PM, Guido van Rossum wrote:
> On Mon, Aug 21, 2017 at 12:50 PM, Yury Selivanov
> wrote:
>>
>> Few important things (using the current PEP 550 terminology):
>>
>> * ExecutionContext is a *dynamic* stack of LogicalContexts.
>> * LCs do not reference other LCs.
>> * ContextKey.set() can only modify the *top* LC in the stack.
>>
>> If LC is a mutable mapping:
>>
>> # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
>>
>> a.set(c)
>> #LC4 = EC.top()
>> #LC4[a] = c
>>
>> # EC = [LC1, LC2, LC3, LC4({a: c, foo: bar})]
>>
>> If LC are implemented with immutable mappings:
>>
>> # EC = [LC1, LC2, LC3, LC4({a: b, foo: bar})]
>>
>> a.set(c)
>> #LC4 = EC.pop()
>> #LC4_1 = LC4.copy()
>> #LC4_1[a] = c
>> #EC.push(LC4_1)
>>
>> # EC = [LC1, LC2, LC3, LC4_1({a: c, foo: bar})]
>>
>> Any code that uses EC will not see any difference, because it can only
>> work with the top LC.
>
>
> OK, good. This makes more sense, especially if I read "the EC" as shorthand
> for the EC stored in the current thread's per-thread state.
That's exactly what I meant by "the EC".
> The immutable
> mapping (if used) is used for the LC, not for the EC, and in this case
> cloning an EC would simply make a shallow copy of its underlying list --
> whereas without the immutable mapping, cloning the EC would also require
> making shallow copies of each LC. And I guess the linked-list implementation
> (Approach #3 in the PEP) makes EC cloning an O(1) operation.
All correct.
>
> Note that there is a lot of hand-waving and shorthand in this explanation,
> but I think I finally follow the design. It is going to be a big task to
> write this up in a didactic way -- the current PEP needs a fair amount of
> help in that sense.
Elvis Pranskevichus (our current What's New editor and my colleague)
offered me to help with the PEP. He's now working on a partial rewrite.
I've been working on this PEP for about a month now and at this point
it makes it difficult for me to dump this knowledge in a nice and
readable way (in any language that I know, FWIW).
> (If you want to become a better writer, I've recently
> enjoyed reading Steven Pinker's The Sense of Style: The Thinking Person's
> Guide to Writing in the 21st Century. Amongst other fascinating topics, it
> explains why so often what we think is clearly written can cause so much
> confusion.)
Will definitely check it out, thank you!
>
>>
>> Back to generators. Generators have their own empty LCs when created
>> to store their *local* EC modifications.
>>
>> When a generator is *being* iterated, it pushes its LC to the EC. When
>> the iteration step is finished, it pops its LC from the EC. If you
>> have nested generators, they will dynamically build a stack of their
>> LCs while they are iterated.
>>
>> Therefore, generators *naturally* control the stack of EC. We can't
>> execute two generators simultaneously in one thread (we can only
>> iterate them one by one), so the top LC always belongs to the current
>> generator that is being iterated:
>>
>> def nested_gen():
>> # EC = [outer_LC, gen1_LC, nested_gen_LC]
>> yield
>> # EC = [outer_LC, gen1_LC, nested_gen_LC]
>> yield
>>
>> def gen1():
>> # EC = [outer_LC, gen1_LC]
>> n = nested_gen()
>> yield
>> # EC = [outer_LC, gen1_LC]
>> next(n)
>> # EC = [outer_LC, gen1_LC]
>> yield
>> next(n)
>> # EC = [outer_LC, gen1_LC]
>>
>> def gen2():
>> # EC = [outer_LC, gen2_LC]
>> yield
>> # EC = [outer_LC, gen2_LC]
>> yield
>>
>> g1 = gen1()
>> g2 = gen2()
>>
>> next(g1)
>> next(g2)
>> next(g1)
>> next(g2)
>
>
> This, combined with your later clarification:
>
>> In the current version of the PEP, generators are initialized with an
>> empty LogicalContext. When they are being iterated (started or
>> resumed), their LogicalContext is pushed to the EC. When the
>> iteration is stopped (or paused), they pop their LC from the EC.
>
> makes it clear how the proposal works for generators. There's an important
> piece that I hadn't figured out from Nick's generator example, because I had
> mistakenly assumed that something *would* be captured at generator create
> time. It's a reasonable mistake to make,
Yeah, it is very subtle.
>
>>
>> HAMT is a way to efficiently implement immutable mappings with O(log32
>> N) set operation, that's it. If we implement immutable mappings using
>> regular dicts and copy, set() would be O(log N).
>
>
> This sounds like abuse of the O() notation. Mathematically O(log N) and
> O(log32 N) surely must be equivalent, since log32 N is just K*(log N) for
> some constant K (about 0.288539), and the constant disappears in the O(), as
> O(K*f(N)) and O(f(N)) are equivalent. Now, I'm happy to hear that a
> HAMT-based implementation is faster than a dict+copy-based imple
Re: [Python-Dev] PEP 550 v3
On Mon, Aug 21, 2017 at 8:06 PM, Guido van Rossum wrote: [..] >> > OK, this really needs to be made very clear early in the PEP. Maybe this >> > final sentence provides the key requirement: changes outside the >> > generator >> > should make it into the generator when next() is invoked, unless the >> > generator itself has made an override; but changes inside the generator >> > should not leak out through next(). >> >> It's covered here with two examples: >> https://www.python.org/dev/peps/pep-0550/#ec-semantics-for-generators > > > I think what's missing is the fact that this is one of the key motivating > reasons for the design (starting with v2 of the PEP). When I encountered > that section I just skimmed it, assuming it was mostly just showing how to > apply the given semantics to generators. I also note some issues with the > use of tense here -- it's a bit confusing to follow which parts of the text > refer to defects of the current (pre-PEP) situation and which parts refer to > how the proposal would solve these defects. I see. The proposal always uses present tense to describe things it adds, and I now see that this is indeed very confusing. This needs to be fixed. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
Ethan Furman wrote: So I like ExecutionContext for the stack of WhateverWeCallTheOtherContext contexts. But what do we call it? How about ExecutionContextFrame, by analogy with stack/stack frame. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On 22 August 2017 at 09:39, Greg Ewing wrote: > Ethan Furman wrote: >> >> So I like ExecutionContext for the stack of WhateverWeCallTheOtherContext >> contexts. But what do we call it? > > How about ExecutionContextFrame, by analogy with stack/stack frame. My latest suggestion to Yury was to see how the PEP reads with it called ImplicitContext, such that: * the active execution context is a stack of implicit contexts * ContextKey.set() updates the innermost implicit context * Contextkey.get() reads the whole stack of active implicit contexts * by default, generators (both sync and async) would have their own implicit context, but you could make them use the context of method callers by doing "gen.__implicit_context__ = None" * by default, coroutines would use their method caller's context, but async frameworks would make sure to give top-level tasks their own independent contexts That proposal came from an initial attempt at redrafting the Abstract and Rationale sections, where it turns out that one of the things the current version of the PEP is somewhat taking for granted is that the reader already has a particular understanding of the difference between explicit state management (i.e. passing things around as function arguments and instance attributes) and implicit state management (i.e. relying on process globals and thread locals). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3 naming
On 22 August 2017 at 10:02, Greg Ewing wrote: > Yury Selivanov wrote: >> >> I can certainly see how "ContextFrame" can be correct if we think >> about "frame" as a generic term, but in Python, people will >> inadvertently think about a connection with frame objects/stacks. > > Calling it ExecutionContextFrame rather than just ContextFrame > would make it clear that it relates to ExecutionContexts in > particular. Please, no - it's already hard enough to help people internalise sync/async design concepts without also introducing ambiguity into the meaning of terms like locals & frame. Instead, let's leave those as purely referring to their existing always-synchronous concepts and find another suitable term for the dynamically nested read/write mappings making up the ExecutionContext :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Tue, Aug 22, 2017 at 12:44 AM, Yury Selivanov wrote: > On Mon, Aug 21, 2017 at 5:39 PM, Koos Zevenhoven > wrote: > [..] > >> In the current version of the PEP, generators are initialized with an > >> empty LogicalContext. When they are being iterated (started or > >> resumed), their LogicalContext is pushed to the EC. When the > >> iteration is stopped (or paused), they pop their LC from the EC. > >> > > > > Another quick one before I go: Do we really need to push and pop a LC on > > each next() call, even if it most likely will never be touched? > > Yes, otherwise it will be hard to maintain the consistency of the stack. > > There will be an optimization: if the LC is empty, we will push NULL > to the stack, thus avoiding the cost of allocating an object. > > But if LCs are immutable, there needs to be only one empty-LC instance. That would avoid special-casing NULL in code. -- Koos > I measured the overhead -- generators will become 0.5-1% slower in > microbenchmarks, but only when they do pretty much nothing. If a > generator contains more Python code than a bare "yield" expression, > the overhead will be harder to detect. -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v3
On Tue, Aug 22, 2017 at 2:06 AM, Koos Zevenhoven wrote: [..] >> There will be an optimization: if the LC is empty, we will push NULL >> to the stack, thus avoiding the cost of allocating an object. >> > But if LCs are immutable, there needs to be only one empty-LC instance. That > would avoid special-casing NULL in code. Yes, this is true. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
