Re: [Python-ideas] PEP draft: context variables

[Nathaniel Smith]

On Sat, Oct 14, 2017 at 9:53 PM, M.-A. Lemburg  wrote:
> I have a hard time seeing the advantage of having a default
> where the context at the time of execution is dependent on
> where it happens rather than where it's defined.
>
> IMO, the default should be to use the context where the line
> was defined in the code, since that matches the intuitive
> way of writing and defining code.

Of course, that's already the default: it's now regular variables and
function arguments work. The reason we have forms like 'with
decimal.localcontext', 'with numpy.errstate' is to handle the case
where you want the context value to be determined by the runtime
context when it's accessed rather than the static context where it's
accessed. That's literally the whole point.

It's not like this is a new and weird concept in Python either -- e.g.
when you raise an exception, the relevant 'except' block is determined
based on where the 'raise' happens (the runtime stack), not where the
'raise' was written:

try:
def foo():
raise RuntimeError
except RuntimeError:
print("this is not going to execute, because Python doesn't work that way")
foo()

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Nick Coghlan]

On 15 October 2017 at 15:05, Guido van Rossum  wrote:

> I would like to reboot this discussion (again). It feels to me we're
> getting farther and farther from solving any of the problems we might solve.
>
> I think we need to give up on doing anything about generators; the use
> cases point in too many conflicting directions. So we should keep the
> semantics there, and if you don't want your numeric or decimal context to
> leak out of a generator, don't put `yield` inside `with`. (Yury and Stefan
> have both remarked that this is not a problem in practice, given that there
> are no bug reports or StackOverflow questions about this topic.)
>

Let me have another go at building up the PEP 550 generator argument from
first principles.

The behaviour that PEP 550 says *shouldn't* change is the semantic
equivalence of the following code:

# Iterator form
class ResultsIterator:
def __init__(self, data):
self._itr = iter(data)
def __next__(self):
return calculate_result(next(self._itr))

results = _ResultsIterator(data)

# Generator form
def _results_gen(data):
for item in data:
yield calculate_result(item)

results = _results_gen(data)

This *had* been non-controversial until recently, and I still don't
understand why folks suddenly decided we should bring it into question by
proposing that generators should start implicitly capturing state at
creation time just because it's technically possible for them to do so (yes
we can implicitly change the way all generators work, but no, we can't
implicitly change the way all *iterators* work).

The behaviour that PEP 550 thinks *should* change is for the following code
to become roughly semantically equivalent, given the constraint that the
context manager involved either doesn't manipulate any shared state at all
(already supported), or else only manipulates context variables (the new
part that PEP 550 adds):

# Iterator form
class ResultsIterator:
def __init__(self, data):
self._itr = iter(data)
def __next__(self):
with adjusted_context():
return calculate_result(next(self._itr))

results = _ResultsIterator(data)

# Generator form
def _results_gen(data):
for item in data:
with adjusted_context():
yield calculate_result(item)

results = _results_gen(data)

Today, while these two forms look like they *should* be comparable, they're
not especially close to being semantically equivalent, as there's no
mechanism that allows for implicit context reversion at the yield point in
the generator form.

While I think PEP 550 would still be usable without fixing this
discrepancy, I'd be thoroughly disappointed if the only reason we decided
not to do it was because we couldn't clearly articulate the difference in
reasoning between:

* "Generators currently have no way to reasonably express the equivalent of
having a context-dependent return statement inside a with statement in a
__next__ method implementation, so let's define one" (aka "context variable
changes shouldn't leak out of generators, as that will make them *more*
like explicit iterator __next__ methods"); and
* "Generator functions should otherwise continue to be unsurprising
syntactic sugar for objects that implement the regular iterator protocol"
(aka "generators shouldn't implicitly capture their creation context, as
that would make them *less* like explicit iterator __init__ methods").

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[M.-A. Lemburg]

On 15.10.2017 06:39, Nick Coghlan wrote:
> On 15 October 2017 at 05:47, Paul Moore  > wrote:
> 
> On 14 October 2017 at 17:50, Nick Coghlan  > wrote:
> > If you capture the context eagerly, then there are fewer opportunities 
> to
> > get materially different values from "data = list(iterable)" and "data =
> > iter(context_capturing_iterable)".
> >
> > While that's a valid intent for folks to want to be able to express, I
> > personally think it would be more clearly requested via an expression 
> like
> > "data = iter_in_context(iterable)" rather than having it be implicit in 
> the
> > way generators work (especially since having eager context capture be
> > generator-only behaviour would create an odd discrepancy between 
> generators
> > and other iterators like those in itertools).
> 
> OK. I understand the point here - but I'm not sure I see the practical
> use case for iter_in_context. When would something like that be used?
> 
> 
> Suppose you have some existing code that looks like this:
> 
> results = [calculate_result(a, b) for a, b in data]
> 
> If calculate_result is context dependent in some way (e.g. a & b might
> be decimal values), then eager evaluation of "calculate_result(a, b)"
> will use the context that's in effect on this line for every result.
> 
> Now, suppose you want to change the code to use lazy evaluation, so that
> you don't need to bother calculating any results you don't actually use:
> 
> results = (calculate_result(a, b) for a, b in data)
> 
> In a PEP 550 world, this refactoring now has a side-effect that goes
> beyond simply delaying the calculation: since "calculate_result(a, b)"
> is no longer executed immediately, it will default to using whatever
> execution context is in effect when it actually does get executed, *not*
> the one that's in effect on this line.
> 
> A context capturing helper for iterators would let you decide whether or
> not that's what you actually wanted by instead writing:
> 
> results = iter_in_context(calculate_result(a, b) for a, b in data)
> 
> Here, "iter_in_context" would indicate explicitly to the reader that
> whenever another item is taken from this iterator, the execution context
> is going to be temporarily reset back to the way it was on this line.
> And since it would be a protocol based iterator-in-iterator-out
> function, you could wrap it around *any* iterator, not just
> generator-iterator objects.

I have a hard time seeing the advantage of having a default
where the context at the time of execution is dependent on
where it happens rather than where it's defined.

IMO, the default should be to use the context where the line
was defined in the code, since that matches the intuitive
way of writing and defining code.

The behavior of also deferring the context to time of
execution should be the non-standard form to not break
this intuition, otherwise debugging will be a pain and
writing fully working code would be really hard in the
face of changing contexts (e.g. say decimal rounding
changes in different parts of the code).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Oct 15 2017)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
  http://www.malemburg.com/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Nick Coghlan]

On 15 October 2017 at 14:53, M.-A. Lemburg  wrote:

> On 15.10.2017 06:39, Nick Coghlan wrote:
> > On 15 October 2017 at 05:47, Paul Moore  > > wrote:
> >
> > On 14 October 2017 at 17:50, Nick Coghlan  > > wrote:
> > > If you capture the context eagerly, then there are fewer
> opportunities to
> > > get materially different values from "data = list(iterable)" and
> "data =
> > > iter(context_capturing_iterable)".
> > >
> > > While that's a valid intent for folks to want to be able to
> express, I
> > > personally think it would be more clearly requested via an
> expression like
> > > "data = iter_in_context(iterable)" rather than having it be
> implicit in the
> > > way generators work (especially since having eager context capture
> be
> > > generator-only behaviour would create an odd discrepancy between
> generators
> > > and other iterators like those in itertools).
> >
> > OK. I understand the point here - but I'm not sure I see the
> practical
> > use case for iter_in_context. When would something like that be used?
> >
> >
> > Suppose you have some existing code that looks like this:
> >
> > results = [calculate_result(a, b) for a, b in data]
> >
> > If calculate_result is context dependent in some way (e.g. a & b might
> > be decimal values), then eager evaluation of "calculate_result(a, b)"
> > will use the context that's in effect on this line for every result.
> >
> > Now, suppose you want to change the code to use lazy evaluation, so that
> > you don't need to bother calculating any results you don't actually use:
> >
> > results = (calculate_result(a, b) for a, b in data)
> >
> > In a PEP 550 world, this refactoring now has a side-effect that goes
> > beyond simply delaying the calculation: since "calculate_result(a, b)"
> > is no longer executed immediately, it will default to using whatever
> > execution context is in effect when it actually does get executed, *not*
> > the one that's in effect on this line.
> >
> > A context capturing helper for iterators would let you decide whether or
> > not that's what you actually wanted by instead writing:
> >
> > results = iter_in_context(calculate_result(a, b) for a, b in data)
> >
> > Here, "iter_in_context" would indicate explicitly to the reader that
> > whenever another item is taken from this iterator, the execution context
> > is going to be temporarily reset back to the way it was on this line.
> > And since it would be a protocol based iterator-in-iterator-out
> > function, you could wrap it around *any* iterator, not just
> > generator-iterator objects.
>
> I have a hard time seeing the advantage of having a default
> where the context at the time of execution is dependent on
> where it happens rather than where it's defined.
>

The underlying rationale is that the generator form should continue to be
as close as we can reasonably make it to being pure syntactic sugar for the
iterator form:

class ResultsIterator:
def __init__(self, data):
self._itr = iter(data)
def __next__(self):
return calculate_result(next(self._itr))

results = _ResultsIterator(data)

The logical context adjustments in PEP 550 then serve to make using a with
statement around a yield expression in a generator closer in meaning to
using one around a return statement in a __next__ method implementation.


> IMO, the default should be to use the context where the line
> was defined in the code, since that matches the intuitive
> way of writing and defining code.
>

This would introduce a major behavioural discrepancy between generators and
iterators.


> The behavior of also deferring the context to time of
> execution should be the non-standard form to not break
> this intuition, otherwise debugging will be a pain and
> writing fully working code would be really hard in the
> face of changing contexts (e.g. say decimal rounding
> changes in different parts of the code).
>

No, it really wouldn't, since "the execution context is the context that's
active when the code is executed" is relatively easy to understand based
entirely on the way functions, methods, and other forms of delayed
execution work (including iterators).

"The execution context is the context that's active when the code is
executed, *unless* the code is in a generator, in which case, it's the
context that was active when the generator-iterator was instantiated" is
harder to follow.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Guido van Rossum]

I would like to reboot this discussion (again). It feels to me we're
getting farther and farther from solving any of the problems we might solve.

I think we need to give up on doing anything about generators; the use
cases point in too many conflicting directions. So we should keep the
semantics there, and if you don't want your numeric or decimal context to
leak out of a generator, don't put `yield` inside `with`. (Yury and Stefan
have both remarked that this is not a problem in practice, given that there
are no bug reports or StackOverflow questions about this topic.)

Nobody understands async generators, so let's not worry about them.

That leaves coroutines (`async def` and `await`). It looks like we don't
want to change the original semantics here either, *except* when a
framework like asyncio or Twisted has some kind of abstraction for a
"task". (I intentionally don't define tasks, but task switches should be
explicit, e.g. via `await` or some API -- note that even gevent qualifies,
since it only switches when you make a blocking call.)

The key things we want then are (a) an interface to get and set context
variables whose API is independent from the framework in use (if any), and
(b) a way for a framework to decide when context variables are copied,
shared or reinitialized.

For (a) I like the API from PEP 550:

var = contextvars.ContextVar('description')
value = var.get()
var.set(value)

It should be easy to adopt this e.g. in the decimal module instead of the
current approach based on thread-local state.

For (b) I am leaning towards something simple that emulates thread-local
state. Let's define "context" as a mutable mapping whose keys are
ContextVar objects, tied to the current thread (each Python thread knows
about exactly one context, which is deemed the current context). A
framework can decide to clone the current context and assign it to a new
task, or initialize a fresh context, etc. The one key feature we want here
is that the right thing happens when we switch tasks via `await`, just as
the right thing happens when we switch threads. (When a framework uses some
other API to switch tasks, the framework do what it pleases.)

I don't have a complete design, but I don't want chained lookups, and I
don't want to obsess over performance. (I would be fine with some kind of
copy-on-write implementation, and switching out the current context should
be fast.) I also don't want to obsess over API abstraction. Finally I don't
want the design to be closely tied to `with`.

Maybe I need to write my own PEP?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Nick Coghlan]

On 15 October 2017 at 05:47, Paul Moore  wrote:

> On 14 October 2017 at 17:50, Nick Coghlan  wrote:
> > If you capture the context eagerly, then there are fewer opportunities to
> > get materially different values from "data = list(iterable)" and "data =
> > iter(context_capturing_iterable)".
> >
> > While that's a valid intent for folks to want to be able to express, I
> > personally think it would be more clearly requested via an expression
> like
> > "data = iter_in_context(iterable)" rather than having it be implicit in
> the
> > way generators work (especially since having eager context capture be
> > generator-only behaviour would create an odd discrepancy between
> generators
> > and other iterators like those in itertools).
>
> OK. I understand the point here - but I'm not sure I see the practical
> use case for iter_in_context. When would something like that be used?
>

Suppose you have some existing code that looks like this:

results = [calculate_result(a, b) for a, b in data]

If calculate_result is context dependent in some way (e.g. a & b might be
decimal values), then eager evaluation of "calculate_result(a, b)" will use
the context that's in effect on this line for every result.

Now, suppose you want to change the code to use lazy evaluation, so that
you don't need to bother calculating any results you don't actually use:

results = (calculate_result(a, b) for a, b in data)

In a PEP 550 world, this refactoring now has a side-effect that goes beyond
simply delaying the calculation: since "calculate_result(a, b)" is no
longer executed immediately, it will default to using whatever execution
context is in effect when it actually does get executed, *not* the one
that's in effect on this line.

A context capturing helper for iterators would let you decide whether or
not that's what you actually wanted by instead writing:

results = iter_in_context(calculate_result(a, b) for a, b in data)

Here, "iter_in_context" would indicate explicitly to the reader that
whenever another item is taken from this iterator, the execution context is
going to be temporarily reset back to the way it was on this line. And
since it would be a protocol based iterator-in-iterator-out function, you
could wrap it around *any* iterator, not just generator-iterator objects.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Paul Moore]

On 14 October 2017 at 17:50, Nick Coghlan  wrote:
> On 14 October 2017 at 21:56, Paul Moore  wrote:
>
> TL;DR of below: PEP 550 currently gives you what you're after, so your
> perspective counts as a preference for "please don't eagerly capture the
> creation time context in generators or coroutines".

Thank you. That satisfies my concerns pretty well.

> The suggestion has been made that we should instead be capturing the active
> context when "url_get(url)" is called, and implicitly switching back to that
> at the point where await is called. It doesn't seem like a good idea to me,
> as it breaks the "top to bottom" mental model of code execution (since the
> "await cr" expression would briefly switch the context back to the one that
> was in effect on the "cr = url_get(url)" line without even a nested suite to
> indicate that we may be adjusting the order of code execution).

OK. Then I think that's a bad idea - and anyone proposing it probably
needs to explain much more clearly why it might be a good idea to jump
around in the timeline like that.

> If you capture the context eagerly, then there are fewer opportunities to
> get materially different values from "data = list(iterable)" and "data =
> iter(context_capturing_iterable)".
>
> While that's a valid intent for folks to want to be able to express, I
> personally think it would be more clearly requested via an expression like
> "data = iter_in_context(iterable)" rather than having it be implicit in the
> way generators work (especially since having eager context capture be
> generator-only behaviour would create an odd discrepancy between generators
> and other iterators like those in itertools).

OK. I understand the point here - but I'm not sure I see the practical
use case for iter_in_context. When would something like that be used?

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Nick Coghlan]

On 14 October 2017 at 21:56, Paul Moore  wrote:

TL;DR of below: PEP 550 currently gives you what you're after, so your
perspective counts as a preference for "please don't eagerly capture the
creation time context in generators or coroutines".

To give an example:
>
> async def get_webpage(id):
> url = f"https://{server}/{app}/items?id={id}";
> # 1
> encoding, content = await url_get(url)
> #2
> return content.decode(encoding)
>
> I would expect that, if I set a context variable at #1, and read it at #2,
> then:
>
> 1. code run as part of url_get would see the value set at 1
> 2. code run as part of url_get could set the value, and I'd see
> the new value at 2
>

This is consistent with what PEP 550 currently proposes, because you're
creating the coroutine and calling it in the same expression: "await
url_get(url)".

That's the same as what happens for synchronous function calls, which is
why we think it's also the right way for coroutines to behave.

The slightly more open-to-challenge case is this one:

# Point 1 (pre-create)
cr = url_get(url)
# Point 2 (post-create, pre-call)
encoding, content = await cr
# Point 3 (post-call)

PEP 550 currently says that it doesn't matter whether you change the
context at point 1 or point 2, as "get_url" will see the context as it is
at the await call (i.e. when it actually gets executed), *not* as it is
when the coroutine is created.

The suggestion has been made that we should instead be capturing the active
context when "url_get(url)" is called, and implicitly switching back to
that at the point where await is called. It doesn't seem like a good idea
to me, as it breaks the "top to bottom" mental model of code execution
(since the "await cr" expression would briefly switch the context back to
the one that was in effect on the "cr = url_get(url)" line without even a
nested suite to indicate that we may be adjusting the order of code
execution).

It would also cause problems with allowing context changes to propagate out
of the "await cr" call, since capturing a context implies forking it, and
hence any changes would somehow need to be transplanted back to a
potentially divergent context history (if a context change *did* happen at
point 2 in the split example).

It doesn't matter what form the lines in the function take (loops,
> with statements, conditionals, ...) as long as they are run
> immediately (class and function definitions should be ignored -
> there's no lexical capture of context variables). That probably means
> "synchronous call stack" in your terms, but please don't assume that
> any implications of that term which aren't covered by the above
> example are obvious to me.
>

I think you got everything, as I really do just mean the stack of frames in
the current thread that will show up in a traceback. We normally just call
it "the call stack", but that's ambiguous whenever we're also talking about
coroutines (since each await chain creates its own distinct asynchronous
call stack).


> > g = gen()
> > with decimal.localcontext() as ctx:
> > ctx.prec = 30
> > for i in g:
> >   pass
>
> "for i in g" is getting values from the generator, at a time when the
> precision is 30, so those values should have precision 30.
>
> There's no confusion here to me. If that's not what decimal currently
> does, I'd happily report that as a bug.
>

This is the existing behaviour that PEP 550 is recommending we preserve as
the default generator semantics, even if decimal (or a comparable context
manager) switches to using context vars instead of thread locals.

As with coroutines, the question has been asked whether or not the "g =
gen()" line should be implicitly capturing the active execution context at
that point, and then switching backing it for each iteration of "for i in
g:".

> The reason I ask that is because there are three "interesting" times in
> the
> > life of a coroutine or generator:
> >
> > - definition time (when the def statement runs - this determines the
> lexical
> > closure)
> > - instance creation time (when the generator-iterator or coroutine is
> > instantiated)
> > - execution time (when the frame actually starts running - this
> determines
> > the runtime call stack)
>
> OK. They aren't *really* interesting to me (they are a low-level
> detail, but they should work to support intuitive semantics, not to
> define what my intuition should be) but I'd say that my expectation is
> that the *execution time* value of the context variable is what I'd
> expect to get and set.
>

That's the view PEP 550 currently takes as well.


> > For asynchronous operations, there's more of a question, since actual
> > execution is deferred until you call await or next() - the original
> > synchronous call to the factory function instantiates an object, it
> doesn't
> > actually *do* anything.
>
> This isn't particularly a question for me: g = gen() creates an
> o

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Antoine Pitrou]

On Sun, 15 Oct 2017 01:46:50 +1000
Nick Coghlan  wrote:
> 
> Might it make more sense to have a parallel *module* that works with a
> different base data type rather than parallel functions within the existing
> API?
> 
> That is, if folks wanted to switch to 64-bit nanosecond time, they would
> use:
> 
> * time_ns.time()
> * time_ns.monotonic()
> * time_ns.perf_counter()
> * time_ns.clock_gettime()
> * time_ns.clock_settime()
> 
> The idea here would be akin to the fact we have both math and cmath as
> modules, where the common APIs conceptually implement the same algorithms,
> they just work with a different numeric type (floats vs complex numbers).

-1 from me.  The math/cmath separation isn't even very well grounded, it
just mirrors the C API that those two modules reflect.  But regardless,
the *operations* in math and cmath are different and operate in
different domains (try e.g. ``sqrt(-1)``), which is not the case here.

Regards

Antoine.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Nick Coghlan]

On 14 October 2017 at 18:21, Antoine Pitrou  wrote:

> On Sat, 14 Oct 2017 10:49:11 +0300
> Serhiy Storchaka 
> wrote:
> > I don't like the idea of adding a parallel set of functions.
> >
> > In the list of alternatives in PEP 410 there is no an idea about fixed
> > precision float type with nanoseconds precision. It can be implemented
> > internally as a 64-bit integer, but provide all methods required for
> > float-compatible number. It would be simpler and faster than general
> > Decimal.
>
> I agree a parallel set of functions is not ideal, but I think a parallel
> set of functions is still more appropriate than a new number type
> specific to the time module.
>
> Also, if you change existing functions to return a new type, you risk
> breaking compatibility even if you are very careful about designing the
> new type.
>

Might it make more sense to have a parallel *module* that works with a
different base data type rather than parallel functions within the existing
API?

That is, if folks wanted to switch to 64-bit nanosecond time, they would
use:

* time_ns.time()
* time_ns.monotonic()
* time_ns.perf_counter()
* time_ns.clock_gettime()
* time_ns.clock_settime()

The idea here would be akin to the fact we have both math and cmath as
modules, where the common APIs conceptually implement the same algorithms,
they just work with a different numeric type (floats vs complex numbers).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Paul Moore]

On 14 October 2017 at 08:09, Nick Coghlan  wrote:
> To try and bring this back to synchronous examples that folks may find more
> intuitive, I figure it's worth framing the question this way: do we want
> people to reason about context variables like the active context is
> implicitly linked to the synchronous call stack, or do we want to encourage
> them to learn to reason about them more like they're a new kind of closure?

I'm really struggling to keep up here. I need to go and fully read the
PEP as Yury suggested, and focus on what's in there. But I'll try to
answer this comment. I will ask one question, though, based on Yury's
point "the PEP is where you should look for the actual semantics" -
can you state where in the PEP is affected by the answer to this
question? I want to make sure that when I read the PEP, I don't miss
the place that this whole discussion thread is about...

I don't think of contexts in terms of *either* the "synchronous call
stack" (which, by the way, is much too technical a term to make sense
to the "non-expert" people around here like me - I know what the term
means, but only in a way that's far to low-level to give me an
intuitive sense of what contexts are) or closures.

At the risk of using another analogy that's unfamiliar to a lot of
people, I think of them in terms of Lisp's dynamic variables. Code
that needs a context variable, gets the value that's current *at that
time*. I don't want to have to think lower level than that - if I have
to, then in my view there's a problem with a *different* abstraction
(specifically async ;-))

To give an example:

async def get_webpage(id):
url = f"https://{server}/{app}/items?id={id}";
# 1
encoding, content = await url_get(url)
#2
return content.decode(encoding)

I would expect that, if I set a context variable at #1, and read it at #2, then:

1. code run as part of url_get would see the value set at 1
2. code run as part of url_get could set the value, and I'd see
the new value at 2

It doesn't matter what form the lines in the function take (loops,
with statements, conditionals, ...) as long as they are run
immediately (class and function definitions should be ignored -
there's no lexical capture of context variables). That probably means
"synchronous call stack" in your terms, but please don't assume that
any implications of that term which aren't covered by the above
example are obvious to me.

To use the decimal context example:

> with decimal.localcontext() as ctx:
> ctx.prec = 30
> for i in gen():
>pass

There's only one setting of a context here, so it's obvious - values
returned from gen have precision 30.

> g = gen()
> with decimal.localcontext() as ctx:
> ctx.prec = 30
> for i in g:
>   pass

"for i in g" is getting values from the generator, at a time when the
precision is 30, so those values should have precision 30.

There's no confusion here to me. If that's not what decimal currently
does, I'd happily report that as a bug.

The refactoring case is similarly obvious to me:

>async def original_async_function():
> with some_context():
> do_some_setup()
> raw_data = await some_operation()
> data = do_some_postprocessing(raw_data)
>
> Refactored:
>
>async def async_helper_function():
> do_some_setup()
> raw_data = await some_operation()
> return do_some_postprocessing(raw_data)
>
>async def refactored_async_function():
> with some_context():
> data = await async_helper_function()
>

All we've done here is take some code out of the with block and write
it as a helper. There should be no change of semantics when doing so.
That's a fundamental principle to me, and honestly I don't see it as
credible for anyone to say otherwise. (Anyone who suggests that is
basically saying "if you use async, common sense goes out of the
window" as far as I'm concerned).

> The reason I ask that is because there are three "interesting" times in the
> life of a coroutine or generator:
>
> - definition time (when the def statement runs - this determines the lexical
> closure)
> - instance creation time (when the generator-iterator or coroutine is
> instantiated)
> - execution time (when the frame actually starts running - this determines
> the runtime call stack)

OK. They aren't *really* interesting to me (they are a low-level
detail, but they should work to support intuitive semantics, not to
define what my intuition should be) but I'd say that my expectation is
that the *execution time* value of the context variable is what I'd
expect to get and set.

> For synchronous functions, instance creation time and execution time are
> intrinsically linked, since the execution frame is allocated and executed
> directly as part of calling the function.
>
> For asynchronous operations, there's more of a question, since actual
> execution

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Antoine Pitrou]

On Sat, 14 Oct 2017 10:49:11 +0300
Serhiy Storchaka 
wrote:

> 13.10.17 17:12, Victor Stinner пише:
> > I would like to add new functions to return time as a number of
> > nanosecond (Python int), especially time.time_ns().
> > 
> > It would enhance the time.time() clock resolution. In my experience,
> > it decreases the minimum non-zero delta between two clock by 3 times,
> > new "ns" clock versus current clock: 84 ns (2.8x better) vs 239 ns on
> > Linux, and 318 us (2.8x better) vs 894 us on Windows, measured in
> > Python.
> > 
> > The question of this long email is if it's worth it to add more "_ns"
> > time functions than just time.time_ns()?
> > 
> > I would like to add:
> > 
> > * time.time_ns()
> > * time.monotonic_ns()
> > * time.perf_counter_ns()
> > * time.clock_gettime_ns()
> > * time.clock_settime_ns()
> > 
> > time(), monotonic() and perf_counter() clocks are the 3 most common
> > clocks and users use them to get the best available clock resolution.
> > clock_gettime/settime() are the generic UNIX API to access these
> > clocks and so should also be enhanced to get nanosecond resolution.  
> 
> I don't like the idea of adding a parallel set of functions.
> 
> In the list of alternatives in PEP 410 there is no an idea about fixed 
> precision float type with nanoseconds precision. It can be implemented 
> internally as a 64-bit integer, but provide all methods required for 
> float-compatible number. It would be simpler and faster than general 
> Decimal.

I agree a parallel set of functions is not ideal, but I think a parallel
set of functions is still more appropriate than a new number type
specific to the time module.

Also, if you change existing functions to return a new type, you risk
breaking compatibility even if you are very careful about designing the
new type.

Regards

Antoine.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Serhiy Storchaka]

13.10.17 17:12, Victor Stinner пише:

I would like to add new functions to return time as a number of
nanosecond (Python int), especially time.time_ns().

It would enhance the time.time() clock resolution. In my experience,
it decreases the minimum non-zero delta between two clock by 3 times,
new "ns" clock versus current clock: 84 ns (2.8x better) vs 239 ns on
Linux, and 318 us (2.8x better) vs 894 us on Windows, measured in
Python.

The question of this long email is if it's worth it to add more "_ns"
time functions than just time.time_ns()?

I would like to add:

* time.time_ns()
* time.monotonic_ns()
* time.perf_counter_ns()
* time.clock_gettime_ns()
* time.clock_settime_ns()

time(), monotonic() and perf_counter() clocks are the 3 most common
clocks and users use them to get the best available clock resolution.
clock_gettime/settime() are the generic UNIX API to access these
clocks and so should also be enhanced to get nanosecond resolution.


I don't like the idea of adding a parallel set of functions.

In the list of alternatives in PEP 410 there is no an idea about fixed 
precision float type with nanoseconds precision. It can be implemented 
internally as a 64-bit integer, but provide all methods required for 
float-compatible number. It would be simpler and faster than general 
Decimal.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add a module itertools.recipes

[Nick Coghlan]

On 14 October 2017 at 16:06, Antoine Rozo  wrote:

> I am not searching for an external library (as I pointed, there are some
> on PyPI like iterutils or more-itertools).
> My point was that recipes are documented in itertools module, but not
> implemented in standard library, and it would be useful to have them
> available.
>

Not providing the recipes as an importable API is a deliberate design
decision, as what folks often need is code that is
similar-to-but-not-exactly-the-same-as the code in the recipe. If they've
copied the code into their own utility library, then that's not a problem -
they can just edit their version to have the exact semantics they need.

Individual recipes may occasionally get promoted to be part of the module
API, but that only happens on a case by case basis, and requires a
compelling justification for the change ("It's sometimes useful" isn't
compelling enough - we know it's sometimes useful, that's why it's listed
as an example recipe).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Nick Coghlan]

On 14 October 2017 at 08:44, Steve Dower  wrote:

>
> It's not possible to special case __aenter__ and __aexit__ reliably
>> (supporting wrappers, decorators, and possible side effects).
>>
>
> Why not? Can you not add a decorator that sets a flag on the code object
> that means "do not create a new context when called", and then it doesn't
> matter where the call comes from - these functions will always read and
> write to the caller's context. That seems generally useful anyway, and then
> you just say that __aenter__ and __aexit__ are special and always have that
> flag set.
>

One example where giving function names implicit semantic significance
becomes problematic:

async def start_transaction(self):
...

async def end_transaction(self, *exc_details):
...

__aenter__ = start_transaction
__aexit__ = end_transaction

There *are* ways around that (e.g. type.__new__ implicitly wraps
__init_subclass__ with classmethod since it makes no sense as a regular
instance method), but then you still run into problems like this:

async def __aenter__(self):
return await self.start_transaction()

async def __aexit__(self, *exc_details):
return await self.end_transaction(*exc_details)

If coroutines were isolated from their parents by default, then the above
method implementations would be broken, even though the exact same
invocation pattern works fine for synchronous function calls.

To try and bring this back to synchronous examples that folks may find more
intuitive, I figure it's worth framing the question this way: do we want
people to reason about context variables like the active context is
implicitly linked to the synchronous call stack, or do we want to encourage
them to learn to reason about them more like they're a new kind of closure?

The reason I ask that is because there are three "interesting" times in the
life of a coroutine or generator:

- definition time (when the def statement runs - this determines the
lexical closure)
- instance creation time (when the generator-iterator or coroutine is
instantiated)
- execution time (when the frame actually starts running - this determines
the runtime call stack)

For synchronous functions, instance creation time and execution time are
intrinsically linked, since the execution frame is allocated and executed
directly as part of calling the function.

For asynchronous operations, there's more of a question, since actual
execution is deferred until you call await or next() - the original
synchronous call to the factory function instantiates an object, it doesn't
actually *do* anything.

The current position of PEP 550 (which I agree with) is that context
variables should default to being closely associated with the active call
stack (regardless of whether those calls are regular synchronous ones, or
asynchronous ones with await), as this keeps the synchronous and
asynchronous semantics of context variables as close to each other as we
can feasibly make them.

When implicit isolation takes place, it's either to keep concurrently
active logical call stacks isolated from each other (the event loop case),
and else to keep context changes from implicitly leaking *up* a stack (the
generator case), not to keep context changes from propagating *down* a call
stack.

When we do want to prevent downward propagation for some reason, then
that's what "run_in_execution_context" is for: deliberate creation of a new
concurrently active call stack (similar to running something in another
thread to isolate the synchronous call stack).

Don't get me wrong, I'm not opposed to the idea of making it trivial to
define "micro tasks" (iterables that perform a context switch to a
specified execution context every time they retrieve a new value) that can
provide easy execution context isolation without an event loop to manage
it, I just think that would be more appropriate as a wrapper API that can
be placed around any iterable, rather than being baked in as an intrinsic
property of generators.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/