Re: [Async-sig] Inadvertent layering of synchronous code as frameworks adopt asyncio

[Yury Selivanov]

> On Mar 27, 2019, at 1:11 AM, Glyph  wrote:
> 
> 
> 
>> On Mar 26, 2019, at 11:56 AM, Yury Selivanov > <mailto:yseliva...@gmail.com>> wrote:
>> 
>> 
>>> On Mar 25, 2019, at 8:01 PM, Guido van Rossum >> <mailto:gu...@python.org>> wrote:
>>> 
>>> Given PBP, I wonder if we should just relent and have a configurable flag 
>>> (off by default) to allow nested loop invocations (both the same loop and a 
>>> different loop).
>>> 
>> 
>> 
>> I think that if we implement this feature behind a flag then some libraries 
>> will start requiring that flag to be set.  Which will inevitably lead us to 
>> a situation where it's impossible to use asyncio without the flag.  
>> Therefore I suppose we should either just implement this behaviour by 
>> default or defer this to 3.9 or later.
> 
> How do you feel about my proposal of making the "flag" be simply an argument 
> to run_until_complete?  If what you really want to do is start a task or 
> await a future, you should get notified that you're reentrantly blocking; but 
> if you're sure, just pass the arg and be on your way.

I'm not sure how making it an argument solves anything; please help me 
understand. I see two scenarios here:

1. Migrating big codebase to asyncio.  This is something that Ben mentioned and 
I also happen to know that a couple big companies were struggling with that.  
In this case a reentrant event loop can ease the migration pain.  But if it's 
enabled via an argument to run_until_complete/run_forever you will have to 
blindly enable it for your entire source tree in order to take advantage of it. 
 This argument then becomes, effectively, a global flag, and discourages you 
from actually refactoring and fixing your code properly.

2. Jupyter case.  Their users would still struggle with copy/pasting 
asyncio.run()/loop.run_until_complete() and get an error. They will still have 
to add "reentrant=True".  From the standpoint of usability, this new argument 
would not be a significant improvement as I see it.

> 
> If it's a "flag" like an env var or some kind of global switch, then I 
> totally agree with you.
> 
>> I myself am -1 on making 'run_until_complete()' reentrant.  The separation 
>> of async/await code and blocking code is painful enough to some people, 
>> introducing another "hybrid" mode will ultimately do more damage than good.  
>> E.g. it's hard to reason about this even for me: I simply don't know if I 
>> can make uvloop (or asyncio) fully reentrant.
> 
> If uvloop has problems with global state that prevent reentrancy, fine - for 
> the use-cases where you're doing this, you already kind of implicitly don't 
> care about performance; someone can instantiate their own, safe loop.  (If 
> you can't do this with asyncio though I kinda wonder what's going on.)

Both asyncio & uvloop have global state: child processes watchers, global 
system hooks, and signal handlers. 

Re process watchers: asyncio manages that itself via monitoring on SIGCHLD etc, 
and uvloop offloads this problem to libuv entirely.  Both still have weird bugs 
with multiple event loops in the same process losing track of their 
subprocesses.

Re signal handlers: they are globally set both for asyncio & uvloop, and 
running a nested loop with some code that doesn't expect to be run like that 
can create hard to debug problems.  We probably need new signals API in asyncio 
(I quite like the signals API in Trio).

Re global system hooks: hooks to intercept async generators creation/GC are 
global.  Event loops do save/restore them in their various run() methods so it 
shouldn't be a problem, but this is still a piece of global state to know about.

I'm also not entirely sure that it's safe to mix uvloop with vanilla asyncio in 
the same process.

> 
>> In case of Jupyter I don't think it's a good idea for them to advertise 
>> nest_asyncio.  IMHO the right approach would be to encourage library 
>> developers to expose async/await APIs and teach Jupyter users to "await" on 
>> async code directly.
> 
> ✨💖✨
> 
>> The linked Jupyter issue (https://github.com/jupyter/notebook/issues/3397 
>> <https://github.com/jupyter/notebook/issues/3397>) is a good example: 
>> someone tries to call "asyncio.get_event_loop().run_until_complete(foo())" 
>> and the call fails.  Instead of recommending to use "nest_asyncio", Jupyter 
>> REPL could simply catch the error and suggest the user to await "foo()".  We 
>> can make that slightly easier by changing the exception type from 
>> Runti

Re: [Async-sig] Inadvertent layering of synchronous code as frameworks adopt asyncio

[Yury Selivanov]

> On Mar 25, 2019, at 8:01 PM, Guido van Rossum  wrote:
> 
> Given PBP, I wonder if we should just relent and have a configurable flag 
> (off by default) to allow nested loop invocations (both the same loop and a 
> different loop).
> 


I think that if we implement this feature behind a flag then some libraries 
will start requiring that flag to be set.  Which will inevitably lead us to a 
situation where it's impossible to use asyncio without the flag.  Therefore I 
suppose we should either just implement this behaviour by default or defer this 
to 3.9 or later.

I myself am -1 on making 'run_until_complete()' reentrant.  The separation of 
async/await code and blocking code is painful enough to some people, 
introducing another "hybrid" mode will ultimately do more damage than good.  
E.g. it's hard to reason about this even for me: I simply don't know if I can 
make uvloop (or asyncio) fully reentrant.

In case of Jupyter I don't think it's a good idea for them to advertise 
nest_asyncio.  IMHO the right approach would be to encourage library developers 
to expose async/await APIs and teach Jupyter users to "await" on async code 
directly.  

The linked Jupyter issue (https://github.com/jupyter/notebook/issues/3397 
) is a good example: someone 
tries to call "asyncio.get_event_loop().run_until_complete(foo())" and the call 
fails.  Instead of recommending to use "nest_asyncio", Jupyter REPL could 
simply catch the error and suggest the user to await "foo()".  We can make that 
slightly easier by changing the exception type from RuntimeError to 
NestedAsyncioLoopError.  In other words, in the Jupyters case, I think it's a 
UI/UX problem, not an asyncio problem.

Yury___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] killing tasks that won't cancel

[Yury Selivanov]

Thanks for referencing that issue, I'll check it out.  I'm also quite curious 
what Nathaniel thinks about this problem and how he thinks he'll handle it in 
Trio.

Yury

> On Feb 19, 2019, at 3:36 PM, Chris Jerdonek  wrote:
> 
> On Tue, Feb 19, 2019 at 12:27 PM Yury Selivanov  wrote:
>> 
>> FYI Chris has started a parallel discussion on the same topic here: 
>> https://bugs.python.org/issue32363.  Chris, let's keep this discussion in 
>> one place (now it's this list, I guess). It's hard to handle the same 
>> discussion in two different places.  Please don't split discussions like 
>> this.
> 
> My apologies. My first comment was on the tracker, but then I realized
> this is a broader discussion, so I moved to this list. For example, it
> seems related to a discussion happening on the trio tracker re:
> graceful shutdown and "hard" and "soft" cancellation, etc:
> https://github.com/python-trio/trio/issues/147
> I'm not sure how similar or different cancellation is across various
> async frameworks.
> 
> --Chris

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] killing tasks that won't cancel

[Yury Selivanov]

> On Feb 19, 2019, at 3:23 PM, Chris Jerdonek  wrote:
> 
> On Tue, Feb 19, 2019 at 12:10 PM Andrew Svetlov
>  wrote:
>> If the task's function swallows CancelledError exception -- it is a
>> programming error.
> 
> I was asking if there is a way to end such a task. Is there? The only
> approach I can think of without having something like set_exception()
> is to keep calling cancel() in a loop and waiting (but even that can
> fail under certain code), but I'm not sure off-hand if the API
> supports calling cancel() more than once.

Unfortunately asyncio isn't super flexible around "cancellation with a timeout" 
kind of scenarios. The current assumption is that once the cancellation is 
requested, the Task will start cancelling and will do so in a timely manner.  
Imposing a second layer of timeouts on the cancellation process itself isn't 
natively supported.  But to properly address this we don't need a very broadly 
defined Task.set_exception(); we need to rethink the cancellation in asyncio 
(perhaps draw some inspiration from Trio and other frameworks).

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] killing tasks that won't cancel

[Yury Selivanov]

FYI Chris has started a parallel discussion on the same topic here: 
https://bugs.python.org/issue32363.  Chris, let's keep this discussion in one 
place (now it's this list, I guess). It's hard to handle the same discussion in 
two different places.  Please don't split discussions like this.

I'll summarize what I said in the above referenced bpo here:

1. Task.set_result() and Task.set_exception() have never ever worked properly.  
They never actually communicated the set result/exception to the underlying 
coroutine.  The fact that they were exposed at all was a simple oversight.

I can guess how Task.set_exception() can be implemented in theory: the 
exception would be thrown into the wrapped coroutine.  But I don't quite 
understand how Task.set_result() can be implemented at all.

2. Task and coroutine maintain a simple relationship: Task wraps its coroutine. 
 The result of the coroutine is the result of the Task (not the other way 
around).  The Task can request its coroutine to cancel.  The coroutine may 
ignore that request by ignoring the asyncio.CancelledError exception.

If the latter happens, the Task cannot terminate the coroutine, this is by 
design.  Moreover, you can always write

   while True:
  try:
 await asyncio.sleep(1)
  except:
 pass

and then nothing can terminate your coroutine.  IOW, if your code chooses to 
ignore CancelledError the Task can do nothing about it.

3. For proper bi-directional communication between coroutines asyncio has 
queues.  One can easily implement a message queue to implement injection of an 
exception or result into a coroutine.

[Chris]
>  I was asking if there is a way to end such a task. Is there? 

No, there's no way to end tasks like that.  The key question here is: is this a 
theoretical problem you're concerned with?  Or is this something that happens 
in real-world framework/library/code that you're dealing with?

Yury


> On Feb 19, 2019, at 2:53 PM, Chris Jerdonek  wrote:
> 
> I have an asyncio question.
> 
> In Python 3.7, is there a way to reliably end a task after having
> already tried calling cancel() on it and waiting for it to end?
> 
> In Python 3.6, I did this with task.set_exception(), but in 3.7 that
> method was removed.
> 
> --Chris
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async_stagger: Happy Eyeballs implementation in pure asyncio

[Yury Selivanov]

On May 13, 2018, 9:35 AM -0400, Guido van Rossum , wrote:

> Yury,
>
> This looks like good work.Would it make sense to add this to asyncio in 3.8?


Yes, it is solid.  I'd like to see this in asyncio; specifically, I suggest to 
add a keyword-only argument to loop.create_connection & asyncio.open_connection 
to use happy eyeballs (off by default). Exposing the "staggered_race()" helper 
function might also be a good idea, I'm just not super happy with the name.

Twistero, would you be interested in submitting a PR?

Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asyncio loop instrumentation

[Yury Selivanov]

I understand why it could be useful to have this in asyncio. But I'm big -1 on 
rushing this functionality in 3.7.

asyncio is no longer provisional, so we have to be careful when we design new 
APIs for it.

Example: I wanted to add support for Task groups to asyncio. A similar concept 
exists in curio and trio and I like it, it can be a big improvement over 
asyncio.gather. But there are too many caveats about handling multiple 
exceptions properly (MultiError?) and some issues with cancellation. That's why 
I decided that it's safer to prototype TaskGroups in a separate package, than 
to push a poorly thought out new API in 3.7.

Same applies to your proposal. You can easily publish a package on PyPI that 
provides an improved version of asyncio event loop. You won't even need to 
write a lot of code, just overload a few methods.

Yury

Sent from my iPhone

> On Jan 2, 2018, at 8:00 PM, Pau Freixes  wrote:
> 
> Agree, poll_start and poll_end suit much better.
> 
> Thanks for the feedback.
> 
> On Tue, Jan 2, 2018 at 1:34 AM, INADA Naoki  wrote:
> For this proposal [4], POC, I've preferred make a reduced list of events:
> 
> * `loop_start` : Executed when the loop starts for the first time.
> * `tick_start` : Executed when a new loop tick is started.
> * `io_start` : Executed when a new IO process starts.
> * `io_end` : Executed when the IO process ends.
> * `tick_end` : Executed when the loop tick ends.
> * `loop_stop` : Executed when the loop stops.
 
 What do you call a "IO process" in this context?
>>> 
>>> Basically the call to the `select/poll/whatever` syscall that will ask
>>> for read or write to a set of file descriptors.
>> 
>> `select/poll/whatever` syscalls doesn't ask for read or write.
>> It waits for read or write (more accurate, waits for readable or
>> writable state).
>> 
>> So poll_start / poll_end looks better name to me.
>> 
>> INADA Naoki  
>> 
>> 
>>> 
>>> Thanks,
>>> 
>>> --
>>> --pau
>>> ___
>>> Async-sig mailing list
>>> Async-sig@python.org
>>> https://mail.python.org/mailman/listinfo/async-sig
>>> Code of Conduct: https://www.python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> --pau
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asyncio loop instrumentation

[Yury Selivanov]

When PEP 567 is accepted, I plan to implement advanced instrumentation in 
uvloop, to monitor basically all io/callback/loop events. I'm still -1 to do 
this in asyncio at least in 3.7, because i'd like us to have some time to 
experiment with such instrumentation in real production code (preferably at 
scale)

Yury

Sent from my iPhone

> On Dec 31, 2017, at 10:02 PM, Antoine Pitrou  wrote:
> 
> On Sun, 31 Dec 2017 18:32:21 +0100
> Pau Freixes  wrote:
>> 
>> These new implementation of the load method - remember that it returns
>> a load factor between 0.0 and 1.0 that inform you about how bussy is
>> your loop -
> 
> What does it mean exactly? Is it the ratio of CPU time over wall clock
> time?
> 
> Depending on your needs, the `psutil` library (*) and/or the new
> `time.thread_time` function (**) may also help.
> 
> (*) https://psutil.readthedocs.io/en/latest/
> (**) https://docs.python.org/3.7/library/time.html#time.thread_time
> 
>> For this proposal [4], POC, I've preferred make a reduced list of events:
>> 
>> * `loop_start` : Executed when the loop starts for the first time.
>> * `tick_start` : Executed when a new loop tick is started.
>> * `io_start` : Executed when a new IO process starts.
>> * `io_end` : Executed when the IO process ends.
>> * `tick_end` : Executed when the loop tick ends.
>> * `loop_stop` : Executed when the loop stops.
> 
> What do you call a "IO process" in this context?
> 
> Regards
> 
> Antoine.
> 
> 
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] APIs for high-bandwidth large I/O?

[Yury Selivanov]

On Dec 7, 2017, 9:03 PM -0500, Yury Selivanov , wrote:
[..]

> I'm quite happy with these results and Ipropose to implement the get_buffer() 
> API (or its equivalent) in Python 3.7.  I've opened an issue [3] to discuss 
> the implementation details.

Issue: https://bugs.python.org/issue32251

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] APIs for high-bandwidth large I/O?

[Yury Selivanov]

On Dec 7, 2017, 9:03 PM -0500, Yury Selivanov , wrote:

>
> I'm quite happy with these results and Ipropose to implement the get_buffer() 
> API (or its equivalent) in Python 3.7.  I've opened an issue [3] to discuss 
> the implementation details.
>

I apologize for the noise, my email client is acting weird.  Here's the url of 
the CPython issue I created to discuss the proposed design: 
https://bugs.python.org/issue32251.


Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] APIs for high-bandwidth large I/O?

[Yury Selivanov]

Hi Antoine,

Thanks for posting this, and sorry for the delayed reply!

I've known about a possibility to optimize asyncio Protocols for a while.  I 
noticed that `Protocol.data_received()` requires making one extra copy of the 
received data when I was working on the initial version of uvloop.  Back then 
my main priority was to make uvloop fully compatible with asyncio, so I wasn't 
really thinking about improving asyncio design.


Let me explain the current flaw of `Protocol.data_received()` so that other 
people on the list can catch up with the discussion:

1. Currently, when a Transport is reading data, it uses `sock.recv()` call, 
which returns a `bytes` object, which is then pushed to 
`Protocol.data_received()`.  Every time `sock.recv()` is called, a new bytes 
object is allocated.

2. Typically, protocols need to accumulate bytes objects they receive until 
they have enough buffered data to be parsed.  Usually a `deque` is used for 
that, less optimized code just concatenates all bytes objects into one.

3. When enough data is gathered and a protocol message can be parsed out of it, 
usually there's a need to concatenate a few buffers from the `deque` or get a 
slice of the concatenated buffer.  At this point, we've copied the received 
data two times.


I propose to add another Protocol base class to asyncio: BufferedProtocol.  It 
won't have the 'data_received()' method, instead it will have 'get_buffer()' 
and 'buffer_updated(nbytes)' methods:

    class asyncio.BufferedProtocol:

        def get_buffer(self) -> memoryview:
            pass

        def buffer_updated(self, nbytes: int):
            pass

When the protocol's transport is ready to receive data, it will call 
`protocol.get_buffer()`.  The latter must return an object that implements the 
buffer protocol.  The transport will request a writable buffer over the 
returned object and receive data *into* that buffer.

When the `sock.recv_into(buffer)` call is done, 
`protocol.buffer_updated(nbytes)` method will be called.  The number of bytes 
received into the buffer will be passed as a first argument.


I've implemented the proposed design in uvloop (branch 'get_buffer', [1]) and 
adjusted your benchmark [2] to use it.  Here are benchmark results from my 
machine (macOS):

vanilla asyncio: 120-135 Mb/s
uvloop: 320-330 Mb/s
uvloop/get_buffer: 600-650 Mb/s.

The benchmark is quite unstable, but it's clear that Protocol.get_buffer() 
allows to implement framing way more efficiently.


I'm also working on porting asyncpg library to use get_buffer(), as it has a 
fairly good benchmark suite.  So far I'm seeing 5-15% speed boost on all 
benchmarks.  What's more important is that get_buffer() makes asyncpg buffer 
implementation simpler!


I'm quite happy with these results and Ipropose to implement the get_buffer() 
API (or its equivalent) in Python 3.7.  I've opened an issue [3] to discuss the 
implementation details.


[1] https://github.com/MagicStack/uvloop/tree/get_buffer
[2] https://gist.github.com/1st1/1c606e5b83ef0e9c41faf21564d75ad7


Thanks,
Yury

On Oct 18, 2017, 2:31 PM -0400, Antoine Pitrou , wrote:
>
> Hi,
>
> I am currently looking into ways to optimize large data transfers for a
> distributed computing framework
> (https://github.com/dask/distributed/). We are using Tornado but the
> question is more general, as it turns out that certain kinds of API are
> an impediment to such optimizations.
>
> To put things short, there are a couple benchmarks discussed here:
> https://github.com/tornadoweb/tornado/issues/2147#issuecomment-337187960
>
> - for Tornado, this benchmark:
> https://gist.github.com/pitrou/0f772867008d861c4aa2d2d7b846bbf0
> - for asyncio, this benchmark:
> https://gist.github.com/pitrou/719e73c1df51e817d618186833a6e2cc
>
> Both implement a trivial form of framing using the "preferred" APIs of
> each framework (IOStream for Tornado, Protocol for asyncio), and then
> benchmark it over 100 MB frames using a simple echo client/server.
>
> The results (on Python 3.6) are interesting:
> - vanilla asyncio achieves 350 MB/s
> - vanilla Tornado achieves 400 MB/s
> - asyncio + uvloop achieves 600 MB/s
> - an optimized Tornado IOStream with a more sophisticated buffering
> logic (https://github.com/tornadoweb/tornado/pull/2166)
> achieves 700 MB/s
>
> The latter result is especially interesting. uvloop uses hand-crafted
> Cython code + the C libuv library, still, a pure Python version of
> Tornado does better thanks to an improved buffering logic in the
> streaming layer.
>
> Even the Tornado result is not ideal. When profiling, we see that
> 50% of the runtime is actual IO calls (socket.send and socket.recv),
> but the rest is still overhead. Especially, buffering on the read side
> still has costly memory copies (b''.join calls take 22% of the time!).
>
> For a framed layer, you shouldn't need so many copies. Once you've
> read the frame length, you can allocate the frame upfront and read into
> it. It is at odds, howe

Re: [Async-sig] Feedback, loop.load() function

[Yury Selivanov]

Hi Pau,

I personally don't think we need this in asyncio.  While the function has a 
relatively low overhead, it's still an overhead, it's a couple more syscalls on 
each loop iteration, and it's a bit of a technical debt.

The bar for adding new functionality to asyncio is very high, and I don't see a 
good rationale for adding this function.  Is it for debugging purposes?  Or for 
profiling live applications?  If it's the latter, then there are so many other 
things we want to see, and some of them are protocol-specific.

If we want to add some tracing/profiling functions there should be a way to 
disable them, otherwise the performance of event loops like uvloop will 
degrade, and I'm not sure that all of its users want to pay a price for 
something they won't ever be using.  All of this just adds to the complexity.

OTOH, event loops in asyncio are pluggable.  You can just subclass the asyncio 
event loop and add your method. asyncio code base is very stable, so you won't 
need to fix your code frequently.

Thanks,
Yury

On Aug 20, 2017, 7:27 PM -0400, Pau Freixes , wrote:
> Hi,
>
> I have a second implementation of the load function [1], the previous
> one was too much naive. The main changes compared with the previous
> one are:
>
> 1) Uses a decay function instead of a vector of samples.
> 2) The load stands for the percentage of CPU used, giving a global
> view of how many CPU resources are still unused.
> 3) The update is done at each _LOAD_FREQ - default 1 second - without
> using a scheduler callback
> 4) Many corner cases fixed
>
> The performance impact introduced in the default loop implemented with
> Python is approx 3% running a trivial program that does not have an
> application overhead [2]. Therefore, in real applications with at
> least some footprint introduced by the application, this performance
> impact should be negligible.
>
> As an example of how the load method can be used, the following code
> [3] runs the loop using different ratios of coroutines per second,
> where each coroutine has a CPU impact of 0.02. Having a maximum
> throughput expected of 50 coroutines per second. In this example, the
> coroutine asks first for the load of the system before start consuming
> the CPU, if the load is higher than 0.9 the coroutine leaves doing
> nothing.
>
> The following snippet shows the execution output :
>
> Load reached for 10.0 coros/seq: 0.20594804872227002, abandoned 0/100
> reseting load
> Load reached for 20.0 coros/seq: 0.40599215789994814, abandoned 0/200
> reseting load
> Load reached for 40.0 coros/seq: 0.8055964270483202, abandoned 0/400
> reseting load
> Load reached for 80.0 coros/seq: 0.9390106831339007, abandoned 450/800
>
> The program runs as was said different levels of throughput printing
> at the final of each one the load of the system reached, and the
> coroutines that were abandoned vs the overall. As can be seen, once
> the last test begins to run and the load of the system reaches the
> 0.90 it is able to reduce the pressure at the application level, in
> that case just leaving doing nothing but in other environments doing
> the proper fallback. As you can notice the load values seem to be
> aligned with the values expected, having in mind that the maximum
> throughput reached would be 50 coroutines per second.
>
> The current implementation tries to return the load taking into
> account the global CPU resources, therefore other processes impacting
> on the use of the same CPU used by the loop should be considered. It
> would give to the developer a reliable metric that can be used in
> environments where the CPU is shared by other processes.
>
> What is missing? Investigate how difficult will be implemented this
> feature in libuv [4] to make it available in uvloop. Give more
> information about the difference between this implementation vs the
> toobusy one [5]. Test performance impact in real applications.
>
> I will like to get more feedback from you. And, if you believe that
> this implementation has some chances to be part of CPython repo which
> would be the next steps that I should make.
>
>
> [1] 
> https://github.com/pfreixes/cpython/commit/ac07fef5af51746c7311494f21b0f067c772a2bf
> [2] https://gist.github.com/pfreixes/233fd8c6a6ec82f2cde4688a2976bf2d
> [3] https://gist.github.com/pfreixes/fd26c36391b33056b7efd525e4690aef
> [4] http://docs.libuv.org/en/v1.x/
> [5] https://github.com/lloyd/node-toobusy
>
> On Sun, Aug 13, 2017 at 12:54 PM, Pau Freixes  wrote:
> > > It looks like your "load average" is computing something very different 
> > > than
> > > the traditional Unix "load average". If I'm reading right, yours is a
> > > measure of what percentage of the time the loop spent sleeping waiting for
> > > I/O, taken over the last 60 ticks of a 1 second timer (so generally 
> > > slightly
> > > longer than 60 seconds). The traditional Unix load average is an
> > > exponentially weighted moving average of the length of the run queue.
> >

Re: [Async-sig] question re: loop.shutdown_asyncgens()

[Yury Selivanov]

On Jul 28, 2017, 4:57 PM -0400, Chris Jerdonek , 
wrote:

> Thanks! I'll try to find time to propose a PR.
>
> Also, for suggestions around the new API, would you prefer that be posted to 
> PR #465, or can it be done here?

I think we can discuss it here, but up to you.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] question re: loop.shutdown_asyncgens()

[Yury Selivanov]

Thanks,
Yury

On Jul 28, 2017, 4:38 PM -0400, Chris Jerdonek , 
wrote:
> Thanks, Yury. Have you also considered including recommended setup / cleanup 
> boilerplate in a place where it's easy for asyncio users to find, like in the 
> asyncio docs here?
> https://docs.python.org/3/library/asyncio-eventloop.html#run-an-event-loop

Yes, a PR would be welcome!

>
> One example of a Python module using this approach is itertools:
> https://docs.python.org/3/library/itertools.html#itertools-recipes
>
> Currently, even the example snippet provided for loop.shutdown_asyncgens():
> https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.shutdown_asyncgens
> is incomplete because it doesn't execute shutdown_asyncgens() in a 
> try-finally like you do in your latest patch posted on PR #465.
>
> Also, even if run() is added to Python 3.7, Python 3.6 users would still need 
> / benefit from being able to find blessed boilerplate in a central place.

I was going to release a new module on PyPI called "asyncio_next" or something 
with backports (and to experiment with the proposed APIs before 3.7 is out).

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] question re: loop.shutdown_asyncgens()

[Yury Selivanov]

One of the design decisions about `loop.close()` is that it doesn't
do a single event loop iteration, making its behaviour highly predictable.
To make `loop.close()` to run `loop.shutdown_asyncgens()` (which is a
coroutine), we would have needed to change that.

One of the ways we want to mitigate this problem in Python 3.7 is to
add a new function to bootstrap asyncio and run top-level coroutines:
`asyncio.run()`.  You can read more about it here: [1].

I'm working on a new PEP that will summarize asyncio changes in 3.7.
I don't have a concrete ETA for it, but I'll try to get the first draft out
by mid September.

[1] https://github.com/python/asyncio/pull/465

Thanks,
Yury

On Jul 27, 2017, 11:24 PM -0400, Chris Jerdonek , 
wrote:
> I have a question about PEP 525 (Asynchronous Generators) which I'm
> sure has a simple answer, but I didn't see it in the PEP or final
> discussion:
> https://mail.python.org/pipermail/python-dev/2016-September/146265.html
>
> Basically, why is the API such that loop.shutdown_asyncgens() must be
> called manually? For example, why can't it be called automatically as
> part of close(), which seems like it would be a friendlier API and
> more helpful to the common case?
>
> I was trying asynchronous iterators in my code and getting the following 
> error:
>
> Exception ignored in:  Traceback (most recent call last):
> File "/usr/local/lib/python3.6/asyncio/queues.py", line 169, in get
> getter.cancel() # Just in case getter is not done yet.
> File "/usr/local/lib/python3.6/asyncio/base_events.py", line
> 574, in call_soon
> self._check_closed()
> File "/usr/local/lib/python3.6/asyncio/base_events.py", line
> 357, in _check_closed
> raise RuntimeError('Event loop is closed')
> RuntimeError: Event loop is closed
>
> Calling loop.shutdown_asyncgens() made the error go away, but it seems
> a little obscure that by adding an asynchronous iterator somewhere in
> your code, you have to remember to check that that line is present
> before loop.close() is called (and the exception message doesn't
> provide a good hint).
>
> Is there any disadvantage to always calling loop.shutdown_asyncgens()
> (i.e. even if it's not needed)? And why might someone need to call it
> at a different time?
>
> Thanks,
> --Chris
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] no current event loop in thread 'MainThread'

[Yury Selivanov]

Hi,

`asyncio.test_utils` is a set of internal undocumented asyncio test utilities. 
We'll likely move them to 'Lib/test' in Python 3.7. Try to use third-party 
testing packages like asynctest instead.

Thanks,
Yury

On Jul 7, 2017, 11:22 AM -0400, wrote:
>
> asyncio.test_utils
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async testing question

[Yury Selivanov]

> On Jul 1, 2017, at 6:49 AM, Dima Tisnek  wrote:
> 
> There's an academic publication from Microsoft where they built a runtime 
> that would run each test really many times, where scheduler is rigged to 
> order runnable tasks differently on each run. I hope someone rewrites this 
> for asyncio

Do you have a link to the publication?

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async documentation methods

[Yury Selivanov]

Hi Dima,

Have you seen https://github.com/asyncio-docs?  I'm trying to get some work 
going there to improve asyncio docs in 3.7. Will start committing more of my 
time there soon.

Thanks,
Yury

On Jun 30, 2017, 6:11 AM -0400, Dima Tisnek , wrote:
> Hi all,
>
> I'm working to improve async docs, and I wonder if/how async methods
> ought to be marked in the documentation, for example
> library/async-sync.rst:
>
> """ ... It [lock] has two basic methods, `acquire()` and `release()`. ... """
>
> In fact, these methods are not symmetric, the earlier is asynchronous
> and the latter synchronous:
>
> Definitions are `async def acquire()` and `def release()`.
> Likewise user is expected to call `await .acquire()` and `.release()`.
>
> This is user-facing documentation, IMO it should be clearer.
> Although there are examples for this specific case, I'm concerned with
> general documentation best practice.
>
> Should this example read, e.g.:
> * two methods, `async acquire()` and `release()`
> or perhaps
> * two methods, used `await x.acquire()` and `x.release()`
> or something else?
>
> If there's a good example already Python docs or in some 3rd party
> docs, please tell.
>
> Likewise, should there be marks on iterators? async generators? things
> that ought to be used as context managers?
>
> Cheers,
> d.
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Yury Selivanov]

Thanks Guido and Nathaniel. I'll work on a fix.

Yury

> On Jun 26, 2017, at 11:19 PM, Nathaniel Smith  wrote:
> 
> I actually thought that async generators already guarded against this using 
> their ag_running attribute. If I try running Dima's example with 
> async_generator, I get:
> 
> sending user-1
> received user-1
> sending user-2
> sending user-0
> Traceback (most recent call last):
> [...]
> ValueError: async generator already executing
> 
> The relevant code is here:
> https://github.com/njsmith/async_generator/blob/e303e077c9dcb5880c0ce9930d560b282f8288ec/async_generator/impl.py#L273-L279
> 
> But I added this in the first place because I thought it was needed for 
> compatibility with native async generators :-)
> 
> -n
> 
> On Jun 26, 2017 6:54 PM, "Yury Selivanov"  wrote:
> (Posting here, rather than to the issue, because I think this actually needs 
> more exposure).
> 
> I looked at the code (genobject.c) and I think I know what's going on here.  
> Normally, when you work with an asynchronous generator (AG) you interact with 
> it through "asend" or "athrow" *coroutines*.
> 
> Each AG has its own private state, and when you await on "asend" coroutine 
> you are changing that state.  The state changes on each "asend.send" or 
> "asend.throw" call.  The normal relation between AGs and asends is 1 to 1.
> 
>   AG - asend
> 
> However, in your example you change that to 1 to many:
> 
>  asend
> /
>   AG - asend
> \
>  asend
> 
> Both 'ensure_future' and 'gather' will wrap each asend coroutine into an 
> 'asyncio.Task'. And each Task will call "asend.send(None)" right in its 
> '__init__', which changes the underlying *shared* AG instance completely out 
> of order.
> 
> I don't see how this can be fixed (or that it even needs to be fixed), so I 
> propose to simply raise an exception if an AG has more than one asends 
> changing it state *at the same time*.
> 
> Thoughts?
> 
> Yury
> 
> > On Jun 26, 2017, at 12:25 PM, Dima Tisnek  wrote:
> >
> > Hi group,
> >
> > I'm trying to cross-use an sync generator across several async functions.
> > Is it allowed or a completely bad idea? (if so, why?)
> >
> > Here's MRE:
> >
> > import asyncio
> >
> >
> > async def generator():
> >while True:
> >x = yield
> >print("received", x)
> >await asyncio.sleep(0.1)
> >
> >
> > async def user(name, g):
> >print("sending", name)
> >await g.asend(name)
> >
> >
> > async def helper():
> >g = generator()
> >await g.asend(None)
> >
> >await asyncio.gather(*[user(f"user-{x}", g) for x in range(3)])
> >
> >
> > if __name__ == "__main__":
> >asyncio.get_event_loop().run_until_complete(helper())
> >
> >
> > And the output it produces when ran (py3.6.1):
> >
> > sending user-1
> > received user-1
> > sending user-2
> > sending user-0
> > received None
> > received None
> >
> >
> > Where are those None's coming from in the end?
> > Where did "user-0" and "user-1" data go?
> >
> > Is this a bug, or am I hopelessly confused?
> > Thanks!
> > ___
> > Async-sig mailing list
> > Async-sig@python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
> 

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Yury Selivanov]

(Posting here, rather than to the issue, because I think this actually needs 
more exposure).

I looked at the code (genobject.c) and I think I know what's going on here.  
Normally, when you work with an asynchronous generator (AG) you interact with 
it through "asend" or "athrow" *coroutines*.

Each AG has its own private state, and when you await on "asend" coroutine you 
are changing that state.  The state changes on each "asend.send" or 
"asend.throw" call.  The normal relation between AGs and asends is 1 to 1.

  AG - asend

However, in your example you change that to 1 to many:

 asend
/
  AG - asend
\
 asend

Both 'ensure_future' and 'gather' will wrap each asend coroutine into an 
'asyncio.Task'. And each Task will call "asend.send(None)" right in its 
'__init__', which changes the underlying *shared* AG instance completely out of 
order.

I don't see how this can be fixed (or that it even needs to be fixed), so I 
propose to simply raise an exception if an AG has more than one asends changing 
it state *at the same time*.

Thoughts?

Yury

> On Jun 26, 2017, at 12:25 PM, Dima Tisnek  wrote:
> 
> Hi group,
> 
> I'm trying to cross-use an sync generator across several async functions.
> Is it allowed or a completely bad idea? (if so, why?)
> 
> Here's MRE:
> 
> import asyncio
> 
> 
> async def generator():
>while True:
>x = yield
>print("received", x)
>await asyncio.sleep(0.1)
> 
> 
> async def user(name, g):
>print("sending", name)
>await g.asend(name)
> 
> 
> async def helper():
>g = generator()
>await g.asend(None)
> 
>await asyncio.gather(*[user(f"user-{x}", g) for x in range(3)])
> 
> 
> if __name__ == "__main__":
>asyncio.get_event_loop().run_until_complete(helper())
> 
> 
> And the output it produces when ran (py3.6.1):
> 
> sending user-1
> received user-1
> sending user-2
> sending user-0
> received None
> received None
> 
> 
> Where are those None's coming from in the end?
> Where did "user-0" and "user-1" data go?
> 
> Is this a bug, or am I hopelessly confused?
> Thanks!
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Yury Selivanov]

> On Jun 26, 2017, at 1:53 PM, Andrew Svetlov  wrote:
> 
> IIRC gather collects coroutines in arbitrary order, maybe it's the source of 
> misunderstanding?

Yes, but that does not explain "receiving None" messages. Let's move this 
discussion to the bug tracker.

Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] async generator confusion or bug?

[Yury Selivanov]

Hi Dima,

> On Jun 26, 2017, at 12:25 PM, Dima Tisnek  wrote:
> 
> Hi group,
> 
> I'm trying to cross-use an sync generator across several async functions.
> Is it allowed or a completely bad idea? (if so, why?)

It is allowed, but leads to complex code.

> 
> Here's MRE:
> 
> import asyncio
> 
> 
> async def generator():
>while True:
>x = yield
>print("received", x)
>await asyncio.sleep(0.1)
> 
> 
> async def user(name, g):
>print("sending", name)
>await g.asend(name)
> 
> 
> async def helper():
>g = generator()
>await g.asend(None)
> 
>await asyncio.gather(*[user(f"user-{x}", g) for x in range(3)])
> 
> 
> if __name__ == "__main__":
>asyncio.get_event_loop().run_until_complete(helper())
> 
> 
> And the output it produces when ran (py3.6.1):
> 
> sending user-1
> received user-1
> sending user-2
> sending user-0
> received None
> received None
> 
> 
> Where are those None's coming from in the end?
> Where did "user-0" and "user-1" data go?


Interesting.  If I replace "gather" with three consecutive awaits of "asend", 
everything works as expected.  So there's some weird interaction of 
asend/gather, or maybe you did find a bug.  Need more time to investigate.

Would you mind to open an issue on bugs.python?

Thanks,
Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [ann] sphinxcontrib-trio: make sphinx better at documenting functions/methods, esp. for async/await code

[Yury Selivanov]

On May 12, 2017 at 7:10:52 PM, Nathaniel Smith (n...@pobox.com) wrote:
> > - The signatures sphinxcontrib-trio currently renders are
> also valid
> (though unusual), so how do you think it should decide which one
> is
> meant?

Ideally, IMO, it should render it like this:

async with Connection.transaction()
await Connection.transaction()
    docstring

> I can see why your API looks like that, but it is a bit
> clever... as you know my personal style guide is that functions
> should
> either support being called like 'await foo()' or 'foo()' but
> never
> both :-).

Yeah, I’m still not sure if it was a good idea :)  asyncpg uses
this trick for a few APIs.

> - Do you have any examples in mind of sphinx rendering multiple
> signatures for the same function? I'm not sure what that looks
> like.

I *think* I saw Sphinx rendering two function signatures
one on top of another, but I can’t remember where.

Anyways, the extension looks quite useful, I’ll try it out!


Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [ann] sphinxcontrib-trio: make sphinx better at documenting functions/methods, esp. for async/await code

[Yury Selivanov]

I like it!

Do you have support for hybrid iterators/context managers:

  async with con.transaction() as tr
  # or
  tr = await con.transaction()

and

  async for row in con.cursor(’SELECT …')
  # or
  cur = await con.cursor(‘SELECT …’)

I.e. if I annotate `con.transaction` with both `:async:` and
`:async-with:`, will it render two signatures?

Yury


On May 12, 2017 at 4:24:22 AM, Nathaniel Smith (n...@pobox.com) wrote:
> Hi all,
>
> I just released a new package, sphinxcontrib-trio:
>
> https://sphinxcontrib-trio.readthedocs.io/
>
> It makes it easier to document many kinds of functions/methods in
> sphinx, including async functions, abstract methods, generators, etc.
>
> I originally wrote it for the trio [1] project, hence the name, but
> don't let that put you off -- there's nothing about it that's specific
> to trio, or even to async/await (except that projects that use
> async/await *really need* an extension like this). Really I think this
> extension ought to be a standard feature of sphinx. But in the mean
> time, it's pretty handy.
>
> -n
>
> [1] https://trio.readthedocs.io
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Debugging destroyed, pending tasks

[Yury Selivanov]

> On Feb 2, 2017, at 2:40 PM, Mark E. Haase  wrote:
> 
> Is it possible to see where this task was created, e.g. the line containing 
> asyncio.ensure_future(asyncio.sleep(...)) ? If not, is this a capability that 
> could be added to debug mode? E.g. ensure_future would add its own stack 
> trace to the task object so that it could be retrieved later when a pending 
> task is destroyed.
> 

Yeah, this is doable. I suggest you to open an issue on bugs.python.org.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future

[Yury Selivanov]

> On Nov 18, 2016, at 6:16 PM, Luca Sbardella  wrote:
> 
> Running 3.6-dev from travis, currently b4+.
> No debug mode, that is an order of magnitude slower ;-)

FWIW I found that I can’t really trust the build times on travis for measuring 
any kind of performance regressions.  You don’t know how it’s compiled there, 
and more importantly, if Python 3.6 is compiled with the same settings as 3.5.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future

[Yury Selivanov]

Also, are you using uvloop or vanilla asyncio?  Try to benchmark vanilla first. 
 And if you have time, please try to test different combinations on vanilla 
asyncio:

Python 3.5 + vanilla asyncio
Python 3.6 + vanilla asyncio
Python 3.6 + Py Future + Py Task
Python 3.6 + Py Future + C Task
Python 3.6 + C Future + C Task
Python 3.6 + Py Future + Py Task

Yury


> On Nov 18, 2016, at 6:02 PM, Yury Selivanov  wrote:
> 
> 
>> On Nov 18, 2016, at 5:53 PM, Luca Sbardella  wrote:
>> 
>> But tests taking 1.48 longer to run on average!
>> Anything I should know about 3.6 and performance?
>> 
> 
> 
> That shouldn’t happen.  Are you sure you aren’t running them in debug mode?  
> Try to comment out imports of ‘_asyncio’ in futures.py and tasks.py and run 
> benchmarks in 3.6 to compare Py Futures to C Futures.
> 
> Also, which Python 3.6 version are you using?  Please try to build one from 
> the repo, I’ve fixed a couple of bugs since 3.6b2.
> 
> Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] [python-tulip] Python 3.6b2 will have C implemented Future

[Yury Selivanov]

> On Nov 18, 2016, at 5:53 PM, Luca Sbardella  wrote:
> 
> But tests taking 1.48 longer to run on average!
> Anything I should know about 3.6 and performance?
> 


That shouldn’t happen.  Are you sure you aren’t running them in debug mode?  
Try to comment out imports of ‘_asyncio’ in futures.py and tasks.py and run 
benchmarks in 3.6 to compare Py Futures to C Futures.

Also, which Python 3.6 version are you using?  Please try to build one from the 
repo, I’ve fixed a couple of bugs since 3.6b2.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Adding asyncio.run() function in Python 3.6

[Yury Selivanov]

> On Nov 16, 2016, at 3:35 PM, Nathaniel Smith  wrote:
> 
> What's the use case for the async generator version? Could the yield be 
> replaced by 'await loop.shutting_down()’?

Async generator version (inspired by contextlib.contextmanager decorator) is 
needed in cases where you want loop.run_forever.

The PR originally proposed to add `asyncio.forever()` (which is the same idea 
as `loop.shutting_down()`), but nobody particularly liked it.

A couple of thoughts/reasons:

1. Some pretty intrusive modifications are required to be made in the event 
loop to make it work.  That means all other event loops (including uvloop) will 
have to be modified to support it.  This is the most important reason.

2. `loop.shutting_down()` is a no go because it’s a method on the loop object.  
We can discuss `asyncio.shutting_down`.  The whole point of this discussion is 
to get rid of the event loop.

3. `await forever()` and `await shutting_down()` have a naming issue - both 
look weird:

  async def main():
srv = await asyncio.start_server(…)
try:
   await asyncio.shutting_down()  # or await forever()
finally:
   srv.close()
   await srv.wait_closed()

In the above example, what does the second ‘await’ do?  Will it be resolved 
when the loop is stopped with ‘loop.stop()’? Or when a KeyboardInterrupt 
occurs?  What will happen if you await on it in parallel from 10 different 
coroutines?  It’s just difficult to define a clear semantics of this coroutine.

Using an asynchronous generator for this case is easier in terms of 
implementation and in terms of specifying the execution semantics.  And the 
approach of using generators for such things isn’t new - we have 
contextlib.contextmanager decorator which is quite popular.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

[Async-sig] Adding asyncio.run() function in Python 3.6

[Yury Selivanov]

One of the remaining problems with the event loop in asyncio is 
bootstrapping/finalizing asyncio programs.

Currently, there are two different scenarios:

[1] Running a coroutine:

async def main():
   # your program
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
finally:
loop.close()

[2] Running a server:

loop = asyncio.get_event_loop()
srv = loop.run_until_complete(
  loop.create_server(…))
try:
  loop.run_forever()
finally:
  try:
srv.close()
loop.run_until_complete(srv.wait_closed())
  finally:
loop.close()

Both cases are *incomplete*: they don’t do correct finalization of asynchronous 
generators. To do that we’ll need to add another 1-3 lines of code (extra 
try-finally).

This manual approach is really painful: 

* It makes bootstrapping asyncio code unnecessarily hard.

* It makes the documentation hard to follow.  And we can’t restructure the docs 
to cover the loop only in the advanced section.

* Most of the people will never know about `loop.shutdown_asyncgen` coroutine.  

* We don’t have a place to add debug code to let people know that their asyncio 
program didn’t clean all resources properly (a lot of unordered warnings will 
be spit out instead).

In https://github.com/python/asyncio/pull/465 I propose to add a new function 
to asyncio in Python 3.6: asyncio.run().

The function can either accept a coroutine, which solves [1]:

async def main():
   # your program
asyncio.run(main())

Or it can accept an asynchronous generator, which solves [2]:

async def main():
  srv = await loop.create_server(…))
  try:
yield  # let the loop run forever
  finally:
srv.close()
await srv.wait_closed()

asyncio.run(main())

asyncio.run() solves the following:

* An easy way to start an asyncio program that properly takes care of loop 
instantiation and finalization.

* It looks much better in the docs.  With asyncio.run people don’t need to care 
about the loop at all, most probably will never use it.

* Easier to experiment with asyncio in REPL.

* The loop and asynchronous generators will be cleaned up properly.

* We can add robust debug output to the function, listing the unclosed tasks, 
servers, connections, asynchronous generators etc, helping people with the 
cleanup logic.

* Later, if we need to add more cleanup code to asyncio, we will have a 
function to add the logic to.

I feel that we should add this to asyncio.  One of the arguments against that, 
is that overloading asyncio.run to accept both coroutines and asynchronous 
generators makes the API more complex.  If that’s really the case, we can add 
two functions: asyncio.run(coro) and asyncio.run_forever(async_generator).

Also take a look at https://github.com/python/asyncio/pull/465.

Thanks,
Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world

[Yury Selivanov]

[..]
> Sorry, this was a bit tongue in cheek.  This was something I said to Guido at 
> the *very* beginning of Tulip development, when asked about mistakes Twisted 
> has made: "don't have a global event loop, you'll never get away from it".
> 
> I still think getting rid of a global loop would always be an improvement, 
> although I suspect it's too late at this point.  `await current_event_loop()` 
> might make more sense in Asyncio as that's not really "global", similar to 
> Curio's trap of the same design; however, I assume that this was an 
> intentional design disagreement for a reason and I don't see that reason as 
> having changed (as Yury indicates).

The latest update of get_event_loop is a step in the right direction. At least 
now we can document the best practices:

1. Have one “main” coroutine to bootstrap/run your program;

2. Don’t design APIs that accept the loop parameter; instead design 
coroutine-first APIs and use get_event_loop in your library if you absolutely 
need the loop.

3. I want to add “asyncio.main(coro)” function, which would create the loop, 
run the “coro” coroutine, and correctly clean everything up.

What you propose, IIUC is a step further:

* Deprecate get_event_loop();

* Add “current_event_loop()” coroutine.

This will enforce (1) and (2), making asyncio library devs/users to focus more 
on coroutines and async/await.

Am I understanding this all correctly?

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Some thoughts on asynchronous API design in a post-async/await world

[Yury Selivanov]

> On Nov 7, 2016, at 12:50 PM, Brett Cannon  wrote:
> 
> 
> 
> On Sun, 6 Nov 2016 at 22:41 Glyph Lefkowitz  wrote:
> 
>> On Nov 6, 2016, at 8:20 PM, Brett Cannon  wrote:
>> 
>> For me there are two questions the post raises. One is how do we keep people 
>> from tying themselves to any one event loop?
> 
> Deprecate, then remove, get_event_loop() :-).
> 
> Is there a bug filed for that at https://github.com/python/asyncio?

I don’t think we need to deprecate get_event_loop().  With 
https://github.com/python/asyncio/pull/452 merged in 3.6, get_event_loop 
becomes more predictable.

Now it’s a documentation issue (I’m trying to work on that) to explain asyncio 
users not to use it (and where they *do* need to use it).

I will also open a PR soon to add asyncio.main() function (or asyncio.run()) to 
further simplify working with asyncio & its documentation.  That should make 
get_event_loop to disappear for end users.

Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] SansIO (Was: PEP: asynchronous generators)

[Yury Selivanov]

> On Aug 8, 2016, at 12:16 PM, Cory Benfield  wrote:
> 
> 
>> On 8 Aug 2016, at 17:06, Yury Selivanov  wrote:
>>> 
>>> As to having a PEP or putting something in PEP 8, I feel pretty lukewarm to 
>>> those ideas. If the core Python team was able to legislate good coding 
>>> practices via PEPs I think the world would be a very different place.
>> 
>> What does this mean?  A lot of people, in any serious project at least, 
>> follow PEP 8 and coding with linters is a wide-spread practice.
> 
> Sure, but that only works for things that linters can enforce, and I 
> sincerely doubt that this is one of them.

Correct.  I don’t think it should be PEP 8 either.  I think Guido’s idea on 
including h11 to the stdlib is cool, and that’s the better way to send the 
message.  We can also add a *new* informational PEP instructing people on why 
(and how) they should write IO free protocol implementations.  It would be 
great if you have time to champion such a PEP (I’d be glad to help if needed.)

The other problem with sans-IO approach is that it takes longer to implement 
properly.  You need at least a synchronous and an asynchronous versions to make 
sure that you got the design parts “right”.  

For instance, just few days ago we’ve open sourced a new PostgreSQL driver [1], 
and right now it’s bound to asyncio.  While the foundational layer of the 
protocol is IO free, it would still take me a lot of time to (a) document it, 
(b) to build and test anything but asyncio on top of it.

And, since our driver is far more feature packed (and faster!) than psycopg2, 
I’d love to make a synchronous version of it (but don’t have time atm).  So 
what I was thinking about is a library that would provide a set of base 
Protocol (meta?) classes, one for asyncio, one for sync io, etc.  Then the only 
thing you would need is to mixin your sans-IO protocol to them and write some 
glue code.  Timeouts, cancellations, etc will be already taken care of for you.

This is something that is super hard to make for protocols like HTTP, but 
shouldn’t be a big problem for DB protocols (redis, postgres, etc).  Something 
to think about :)

[1] http://magic.io/blog/asyncpg-1m-rows-from-postgres-to-python/

Thanks,
Yury






> How would you code a linter in order to confirm that protocol code and I/O 
> code are separated?
> 
> More importantly though, people follow the bits of PEP 8 that are easy to 
> enforce and that don’t require substantive architectural changes. If PEP 8 
> said something like “wherever possible use dependency injection as a design 
> pattern”, that guideline would be ignored as both entirely unenforceable and 
> totally subjective. Where is the line for “wherever possible”? How does one 
> enforce the use of dependency injection? Can you programmatically determine 
> dependency injection? What dependencies do not need to be injected?
> 
> Dependency injection is a great design pattern that produces lots of useful 
> side effects, and I use it often. But if I saw PEP 8 mandating it I’d be 
> extremely perplexed. Realistically, at a certain point it’s the equivalent of 
> writing “You should write good, maintainable code, obeying all relevant best 
> practices” into PEP 8. *Of course* you should. This goes without saying. But 
> that makes the advice not that helpful.
> 
> Now, PEP 8 could *recommend* design patterns to follow, and that’s a whole 
> other kettle of fish. But then we’re just asking a different question: how 
> universally praised does a design pattern have to be to become part of PEP 8?
> 
> Cory

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] SansIO (Was: PEP: asynchronous generators)

[Yury Selivanov]

> On Aug 8, 2016, at 11:56 AM, Cory Benfield  wrote:
> 
> 
>> On 8 Aug 2016, at 16:47, Guido van Rossum  wrote:
>> 
>> OK, never mind Impostor Syndrome... How can we move this forward in
>> the community though actual applications? Should we have the complete
>> I/O-free HTTP parser in the stdlib, or is that a bad idea? Do we want
>> a PEP? Is there something we could add to PEP 8?
> 
> Addressing those in turn:
> 
> Yes, we could put the I/O free HTTP parser in the stdlib. That’s really 
> Nathaniel’s call, of course, as it’s his parser, but there’s no reason we 
> couldn’t. Of course, all the regular caveats apply: we’ll want to give it a 
> while longer on PyPI to bake, and of course we have to discuss where the 
> logical end of this. How obscure does a protocol have to be for having a 
> parser in the stdlib to no longer be sensible?
> 
> Having a parser in the stdlib also raises another question: does the http 
> module get rewritten in terms of this parser? The obvious answer is “yes”,

This — we have to validate that h11 has the correct design *before* including 
it to the stdlib.  So the answer is “yes”, indeed.

> 
> As to having a PEP or putting something in PEP 8, I feel pretty lukewarm to 
> those ideas. If the core Python team was able to legislate good coding 
> practices via PEPs I think the world would be a very different place.

What does this mean?  A lot of people, in any serious project at least, follow 
PEP 8 and coding with linters is a wide-spread practice.


Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] PEP: asynchronous generators

[Yury Selivanov]

> To keep it simple, try thinking like this (and yes, Yury, apologies - this is 
> now a side discussion, and not about this pep):  everything in CPython is 
> async, and if you don't want async, you don't need to know about, you run a 
> single async task and don't need to know more...
> 
> Can we get there?
> That would be cool...


So, essentially, something similar to Golang?  I don’t know if that’s possible. 
 It would require a complete CPython IO layer rewrite, integrating an event 
loop directly into the core, etc.  The closest thing to that is gevent — no 
async/await and all IO is non-blocking, but it has its own warts.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] PEP: asynchronous generators

[Yury Selivanov]

[..]
> Currently, I just don't see any other way to use some async generator (that 
> needs some time to be closed) without getting warning.

It will be a very rare case — you need a generator that produces an infinite 
series of values with a try..finally block.

For those kinds of generators you will have a standard asyncio warning that a 
task was GCed in pending state.

We can maintain a weak set of ‘aclose’ tasks in the event loop, and make it a 
public and documented property.  That way you’ll be able to call gather() with 
a timeout on that list before closing the loop.  

The PEP doesn’t specify how exactly asyncio should be modified.  This is 
something we can discuss when the PEP patch is reviewed.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] PEP: asynchronous generators

[Yury Selivanov]

> On Jul 30, 2016, at 3:32 AM, Герасимов Михаил  wrote:
> 
> When asynchronous generator is about to be garbage collected event loop 
> starts task to execute aclose, right? Can't such situation happen when this 
> task is not finished at the moment event loop is closed? Something like:


Yes, such situations can happen.  BUT they can happen for regular coroutines 
too!  Try running the following:

async def coro1():
 try:
 print('try')
 await asyncio.sleep(1)
 finally:
 await asyncio.sleep(0)
 print('finally')
async def coro2():
 await asyncio.sleep(0)
loop = asyncio.get_event_loop()
loop.create_task(coro1())
loop.run_until_complete(coro2())
loop.close()

In the above script, coro1() enters its `try` block and prints “try”; loop ends 
the execution because coro2() is completed; and coro1() will never execute its 
`finally` block.

Long story short - you have to be extra careful when you’re closing the event 
loop.  I believe that not executing “finally” statements when the process is 
about to complete will be more common for coroutines than async generators.

Also, in real life programs you don’t run and close event loops more than once 
per process lifetime.  So it should be fine if you have a few non-exhausted 
generators and coroutines being GCed without proper finalization just before 
the process exits.  And, BTW, Python interpreter will issue a ResourceWarning 
that it cannot finalize coroutines and/or async generators.


Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] PEP: asynchronous generators

[Yury Selivanov]

> On Jul 29, 2016, at 3:06 PM, Andrew Svetlov  wrote:
> 
> I'm personally think that `a` prefix is pretty fine and understandable.
> 
> Do we really need to implement full PEP-342 spec in async way?
> What are use cases?
> My first thought is PEP-255-like simple async generators should be enough.
> 
> Full PEP-342 styled generators are clue for implementing things like 
> twisted.inlineCallback and tornado.gen.
> 
> Thanks to PEP-492 we don't need it anymore.
> 
> All my requirements are covered perfectly fine by existing 
> `__aiter__`/`__anext__`.
> What I really need is allowing to write just `yield val` from coroutine but 
> never need for `resp = yield req`

There are a few reasons for having asend() and athrow():

1) They will have to be implemented regardless; aclose() is a slightly limited 
version of athrow(), and __anext__() is almost the same thing as asend().  It’s 
5-10 additional lines of code to have them.

2) __anext__() has to be a generator-like object, so that YIELD_FROM opcode 
works correctly with it. That means it has to implement ‘send()’ and ‘throw()’ 
methods.  Which, in turn, means that even if we don’t expose ‘agen.athrow()’, 
people would be able to do ‘agen.__anext__().throw()’.  But that’s a very 
error-prone thing to do.

3) Having ‘anext()’ and ‘athrow()’ enable you to implement decorators like 
`@contextlib.contextmanager` but for ‘async with’.  And I’m pretty sure that 
people will come up with even more creative things.

To conclude, having ‘asend()’ and ‘athrow()’ doesn’t complicate the 
implementation at all, but makes asynchronous generators consistent and more 
compatible with synchronous generators, which I don’t think is a bad thing.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] PEP: asynchronous generators

[Yury Selivanov]

> On Jul 29, 2016, at 2:50 PM, Yarko Tymciurak  wrote:
> To keep it simple, try thinking like this (and yes, Yury, apologies - this is 
> now a side discussion, and not about this pep):  everything in CPython is 
> async, and if you don't want async, you don't need to know about, you run a 
> single async task and don't need to know more...
> 
> Can we get there?
> That would be cool...


So something like what they have in Golang?  I don’t know if that’s possible in 
CPython...

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] PEP: asynchronous generators

[Yury Selivanov]

Comments inlined:


> On Jul 29, 2016, at 2:20 PM, Yarko Tymciurak  wrote:
> 
> Hmm...  I think we need to think about a future where, programmatically, 
> there's little-to no distinction between async and synchronous functions. 
> Pushing this down deeper in the system is the way to go. For one, it will 
> serve simple multi-core use once gilectomy is completed (it, or something 
> effectively equivalent will complete).  For another, this is the path to 
> reducing the functionally "useless" rewrite efforts of libraries (e.g. 
> github.com/aio-libs), which somehow resemble all the efforts of migrating 
> libraries from 2-3 (loosely).  The resistance and unexpected time that 2-3 
> migration experienced won't readily be mimicked in async tasks - too much 
> effort to get computer and I/O bound benefits?  Maintain two versions of 
> needed libraries, or jump languages is what will increasingly happen in the 
> distributed (and more so IOT) world.

When and *if* gilectomy is completed (or another project to remove the GIL), we 
will be able to do this:

1) Run existing async/await applications as is, but instead of running a 
process per core, we will be able to run a single process with many threads.  
Likely one asyncio (or other) event loop per thread.  This is very speculative, 
but possible in theory.

2) Run existing blocking IO applications in several threads in one process.  
This is something that only sounds like an easy thing to do, I suspect that a 
lot of code will break (or dead lock) when the GIL is removed.  Even if 
everything works perfectly well, threads aren’t answer to all problems — try to 
manage 1000s of them.

Long story short, even if we had no GIL at all, having async/await (and 
non-blocking IO) would make sense.  And if you have async/await, with GIL or 
without, you will inevitably have different APIs and different IO low-level 
libs that drive them.

There are ways to lessen the pain.  For instance, I like Cory’s approach with 
hyper - implement protocols separately from IO, so that it’s easy to port them 
to various sync and async frameworks.

> 
> Time to think about paving the way to async-as first class citizen world.
> 
> That's probably too much for this PEP, but the topic (a- prefixing) is a good 
> canary for the bigger picture we need to start mulling over.
> 
> So in this context (and in general w/ async) asking the question "can we make 
> it so it doesn't matter?" is a good one to always be asking - it will get is 
> there.
> 

Unfortunately there is no way to use the same APIs for both async/await and 
synchronous world.  At least for CPython builtin types they have to have 
different names.

I’m fine to discuss the ‘a’ prefix, but I’m a bit afraid that focusing on it 
too much will distract us from the PEP and details of it that really matter.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] PEP: asynchronous generators

[Yury Selivanov]

Few comments below:

> On Jul 29, 2016, at 1:57 PM, Brett Cannon  wrote:
> 
> 
> 
> On Fri, 29 Jul 2016 at 10:46 Yury Selivanov  wrote:
> Thanks a lot for the feedback, Brett!  Comments inlined below:
> 
> > On Jul 29, 2016, at 1:25 PM, Brett Cannon  wrote:
> >
> [..]
> >
> > Performance is an additional point for this proposal: in our testing of
> > the reference implementation, asynchronous generators are *2x* faster
> > than an equivalent implemented as an asynchronous iterator.
> >
> > Another motivation is that types.coroutine becomes purely a 
> > backwards-compatibility/low-level thing that is no longer required for 
> > event loop frameworks to use on their generators which provide their async 
> > API. So now an async framework can be written entirely in terms of def and 
> > async def and not def and @types.coroutine.
> 
> A slight misunderstanding here: @types.coroutine turns a normal old-style 
> generator into an awaitable object.  That isn’t directly related to 
> asynchronous iteration.  Frameworks like curio will continue using 
> @types.coroutine to implement future-like objects in event loops.
> 
> Ah, OK. So that would be yet another PEP to make that kind of change (and 
> another keyword).

TBH I’m not really sure we need that.  To separate coroutines from generators 
completely, you don’t just need another keyword — you basically need to have a 
separate parallel implementation of the whole iteration protocol.  I think 
@types.coroutine is a nice glue between two worlds, allowing us to reuse the 
code efficiently.  Again, just my 2 cents.

> >
> > I'm going to ask this now instead of later when there's more motivation 
> > behind this question: do we need to append "a" to every async method we 
> > have? If asynchronous generators won't have a normal close() then why can't 
> > it just be close(), especially if people are not going to be calling it 
> > directly and instead it will be  event loops? I'm just leery of codifying 
> > this practice of prepending "a" to every async method or function and 
> > ending up in a future where I get tired of a specific letter of the 
> > alphabet.
> 
> I decided to use the prefix because we already use it in magic method names: 
> __anext__ and __aiter__.  I think it also makes it easier to understand the 
> API of async generators (and understand how it’s different from sync 
> generators API).
> 
> And while it’s entirely possible to drop the ‘a’ for async generator API, 
> it’s not so simple for other cases.  Later, for Python 3.7, we might consider 
> adding ‘aiter()’ and ‘anext()’ builtins, for which we’d have to use ‘a’ or 
> ‘async’ prefix (we can’t reuse 'iter()' and 'next()’ for async generators).
> 
> I guess we just need to decide as a group that an 'a' prefix is what we want 
> to signify something is asynchronous vs some other prefix like 'a_' or 
> 'async', or 'async_' as people will follow this style choice in their own 
> code going forward.

I’m open to having this discussion.  I don’t have a strong preference here; I, 
personally, like the ‘a’ prefix slightly better, because it’s consistent with 
__a*__ methods and easy to type.

[..]
> We actually added that in 3.5 (last minute!).
> 
> For sync generators the field is called ‘.gi_yieldfrom’, for coroutines it’s 
> ‘.cr_await’, and for proposed async generators it will be ‘.ag_await’.
> 
> I would also clarify that "waiting on" means "what `await` has been called on 
> (if anything as an `await` call might not have been used)" and not what the 
> last yielded object happened to be (which is what my brain initially thought 
> it was simply because the async generator is paused on the event loop 
> returning based on what was yielded).

OK, I’ll try to clarify this!

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] PEP: asynchronous generators

[Yury Selivanov]

Thanks a lot for the feedback, Brett!  Comments inlined below:

> On Jul 29, 2016, at 1:25 PM, Brett Cannon  wrote:
> 
[..]
> 
> Performance is an additional point for this proposal: in our testing of
> the reference implementation, asynchronous generators are *2x* faster
> than an equivalent implemented as an asynchronous iterator.
> 
> Another motivation is that types.coroutine becomes purely a 
> backwards-compatibility/low-level thing that is no longer required for event 
> loop frameworks to use on their generators which provide their async API. So 
> now an async framework can be written entirely in terms of def and async def 
> and not def and @types.coroutine.

A slight misunderstanding here: @types.coroutine turns a normal old-style 
generator into an awaitable object.  That isn’t directly related to 
asynchronous iteration.  Frameworks like curio will continue using 
@types.coroutine to implement future-like objects in event loops.

[..]
> 
> Support for Asynchronous Iteration Protocol
> ---
> 
> The protocol requires two special methods to be implemented:
> 
> 1. An ``__aiter__`` method returning an *asynchronous iterator*.
> 2. An ``__anext__`` method returning an *awaitable* object, which uses
>``StopIteration`` exception to "yield" values, and
>``StopAsyncIteration`` exception to signal the end of the iteration.
> 
> I assume this is in place of __iter__() and __next__(), i.e. an async 
> generator won't have both defined?

Correct, async generators don’t have __iter__ and __next__. They can only be 
iterated with an `async for`.

[..]
> 
> To solve this problem we propose to do the following:
> 
> 1. Implement an ``aclose`` method on asynchronous generators
>returning a special *awaitable*.  When awaited it
>throws a ``GeneratorExit`` into the suspended generator and
>iterates over it until either a ``GeneratorExit`` or
>a ``StopAsyncIteration`` occur.
> 
>This is very similar to what the ``close()`` method does to regular
>Python generators, except that an event loop is required to execute
>``aclose()``.
> 
> I'm going to ask this now instead of later when there's more motivation 
> behind this question: do we need to append "a" to every async method we have? 
> If asynchronous generators won't have a normal close() then why can't it just 
> be close(), especially if people are not going to be calling it directly and 
> instead it will be  event loops? I'm just leery of codifying this practice of 
> prepending "a" to every async method or function and ending up in a future 
> where I get tired of a specific letter of the alphabet.

I decided to use the prefix because we already use it in magic method names: 
__anext__ and __aiter__.  I think it also makes it easier to understand the API 
of async generators (and understand how it’s different from sync generators 
API).

And while it’s entirely possible to drop the ‘a’ for async generator API, it’s 
not so simple for other cases.  Later, for Python 3.7, we might consider adding 
‘aiter()’ and ‘anext()’ builtins, for which we’d have to use ‘a’ or ‘async’ 
prefix (we can’t reuse 'iter()' and 'next()’ for async generators).

[..]

> 
> 3. ``agen.anext(val)``: Returns an *awaitable*, that pushes the
>``val`` object in the ``agen`` generator.  When the ``agen`` has
>not yet been iterated, ``val`` must be ``None``.
> 
> How is this different from an async send()? I see an asend() used in the 
> example below but not anext().

Good catch!  This is a typo, it should read ``agen.asend(val)``.

Similarly to sync generators, where ‘__next__()’ call is equivalent of 
‘.send(None)’, ‘__anext__()’ awaitable is equivalent to ‘.asend(None)'

[..]
> 
> 7. ``agen.ag_await``: The object that ``agen`` is currently awaiting on,
>or ``None``.
> 
> That's an interesting addition. I like it! There's no equivalent on normal 
> generators, correct?

We actually added that in 3.5 (last minute!).

For sync generators the field is called ‘.gi_yieldfrom’, for coroutines it’s 
‘.cr_await’, and for proposed async generators it will be ‘.ag_await’.

Thanks!
Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

[Async-sig] PEP: asynchronous generators

[Yury Selivanov]

Hi,

I have been working on a PEP to add asynchronous generators to 
Python.  The PEP is now ready for a review.  It would be great to 
hear some initial feedback from async-sig, before I post it to 
python-ideas.

I have a complete and working reference implementation of 
everything that PEP proposes here: 

https://github.com/1st1/cpython/tree/async_gen


PEP: XXX
Title: Asynchronous Generators
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov 
Discussions-To: 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 28-Jul-2016
Python-Version: 3.6
Post-History:


Abstract


PEP 492 introduced support for native coroutines and ``async``/``await``
syntax to Python 3.5.  It is proposed here to extend Python's
asynchronous capabilities by adding support for
*asynchronous generators*.


Rationale and Goals
===

Regular generators (introduced in PEP 255) enabled an elegant way of
writing complex *data producers* and have them behave like an iterator.

However, currently there is no equivalent concept for the *asynchronous
iteration protocol* (``async for``).  This makes writing asynchronous
data producers unnecessarily complex, as one must define a class that
implements ``__aiter__`` to be able to use it in an ``async for``
statement.

Essentially, the goals and rationale for PEP 255, applied to the
asynchronous execution case, hold true for this proposal as well.

Performance is an additional point for this proposal: in our testing of
the reference implementation, asynchronous generators are *2x* faster
than an equivalent implemented as an asynchronous iterator.

As an illustration of the code quality improvement, consider the
following class that prints numbers with a given delay once iterated::

class ticker:
"""Print numbers from 0 to `to` every `delay` seconds."""

def __init__(self, delay, to):
self.delay = delay
self.i = 0
self.to = to

def __aiter__(self):
return self

async def __anext__(self):
i = self.i
if i >= self.to:
raise StopAsyncIteration
self.i += 1
if i:
await asyncio.sleep(self.delay)
return i


The same can be implemented as a much simpler asynchronous generator::

async def ticker(delay, to):
"""Print numbers from 0 to `to` every `delay` seconds."""
i = 0
while i < to:
yield i
i += 1
await asyncio.sleep(delay)


Specification
=

This proposal introduces the concept of *asynchronous generators* to
Python.

This specification presumes knowledge of the implementation of
generators and coroutines in Python (PEP 342, PEP 380 and PEP 492).


Asynchronous Generators
---

A Python *generator* is any function containing one or more ``yield``
expressions::

def func():# a function
return

def genfunc(): # a generator function
yield

We propose to use the same approach to define
*asynchronous generators*::

async def coro():  # a coroutine function
await smth()

async def asyncgen():  # an asynchronous generator function
await smth()
yield val

The result of calling an *asynchronous generator function* is
an *asynchronous generator object*, which implements the asynchronous
iteration protocol defined in PEP 492.

It is a ``SyntaxError`` to have a non-empty ``return`` statement in an
asynchronous generator.


Support for Asynchronous Iteration Protocol
---

The protocol requires two special methods to be implemented:

1. An ``__aiter__`` method returning an *asynchronous iterator*.
2. An ``__anext__`` method returning an *awaitable* object, which uses
   ``StopIteration`` exception to "yield" values, and
   ``StopAsyncIteration`` exception to signal the end of the iteration.

Asynchronous generators define both of these methods::

async def genfunc():
yield 1
yield 2

gen = genfunc()

assert gen.__aiter__() is gen

assert await gen.__anext__() == 1
assert await gen.__anext__() == 2

with assertRaises(StopAsyncIteration):
await gen.__anext__()


Finalization


PEP 492 requires an event loop or a scheduler to run coroutines.
Because asynchronous generators are meant to be used from coroutines,
they also require an event loop to run and finalize them.

Asynchronous generators can have ``try..finally`` blocks, as well as
``async with``.  It is important to provide a guarantee that, even
when partially iterated, and then garbage collected, generators can
be safely finalized.  For example::

async def square_series(con, to):
async with con.transaction():
cursor = con.cursor(

Re: [Async-sig] Asynchronous cleanup is a problem

[Yury Selivanov]

> On Jul 6, 2016, at 9:44 PM, Nathaniel Smith  wrote:
> 
> On Wed, Jul 6, 2016 at 6:17 PM, Yury Selivanov  wrote:
>> 
>>> ...does it actually work to re-enter a main loop from inside a __del__
>>> callback? It seems like you could get into really nasty states with
>>> multiple nested __del__ calls, or if a single sweep detects multiple
>>> pieces of garbage with __del__ methods, then some of those __del__
>>> calls could be delayed indefinitely while the first __del__ runs. Is
>>> the cycle collector even re-entrant?
>> 
>> We can have a flag on the async gen object to make sure that we run the 
>> finalizer only once. The finalizer will likely schedule async_gen.aclose() 
>> coroutine which will ensure a strong ref to the gen until it is closed. This 
>> can actually work.. ;)
> 
> Hmm, if the strategy is to schedule the work to happen outside of the
> actual __del__ call, then I think this is back to assuming that all
> coroutine runners are immortal and always running. Is that an
> assumption you're comfortable with?

No need to require coroutine runners to be immortal.

If a finalizer finds out that its event loop is closed, it does nothing.
The interpreter will then issue a ResourceWarning if the generator wasn’t
properly closed.  This is how a finalizer for asyncio event loop might
look like:

def _finalize_gen(self, gen):
if not self.is_closed():
self.create_task(gen.aclose())

And this is how asyncio loop might set it up:

class AsyncioEventLoop:
def run_forever():
...
old_finalizer = sys.get_async_generator_finalizer()
sys.set_async_generator_finalizer(self._finalize_gen)
try:
while True:
self._run_once()
…
finally:
…
sys.set_async_generator_finalizer(old_finalizer)


Why do I think that it’s OK that some async generators might end up 
not being properly closed?  Because this can already happen to 
coroutines:

async def coro1():
try:
print('try')
await asyncio.sleep(1)
finally:
await asyncio.sleep(0)
print('finally')
async def coro2():
await asyncio.sleep(0)
loop = asyncio.get_event_loop()
loop.create_task(coro1())
loop.run_until_complete(coro2())

In other words, an event loop might stop or be interrupted, and there
is no way to guarantee that all coroutines will be finalized properly
in that case.

To address Glyph’s point about many event loops in one process:
a finalizer set with set_async_generator_finalizer() (which should
be thread specific, same as set_coroutine_wrapper) can actually be 
assigned to the async generator when it’s instantiated.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asynchronous cleanup is a problem

[Yury Selivanov]

> ...does it actually work to re-enter a main loop from inside a __del__
> callback? It seems like you could get into really nasty states with
> multiple nested __del__ calls, or if a single sweep detects multiple
> pieces of garbage with __del__ methods, then some of those __del__
> calls could be delayed indefinitely while the first __del__ runs. Is
> the cycle collector even re-entrant?

We can have a flag on the async gen object to make sure that we run the 
finalizer only once. The finalizer will likely schedule async_gen.aclose() 
coroutine which will ensure a strong ref to the gen until it is closed. This 
can actually work.. ;)

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asynchronous cleanup is a problem

[Yury Selivanov]

> On Jul 6, 2016, at 8:47 PM, Glyph Lefkowitz  wrote:
> 
> 
>> On Jul 6, 2016, at 5:25 PM, Yury Selivanov  wrote:
>> 
>> The problem is that the GC can’t execute async code, and we don’t have any 
>> control over GC.  What if we add a mechanism to control how async generators 
>> (AG) are destructed.  Let’s say we add new function to the sys module - 
>> `sys.set_async_generator_finalizer(finalizer)`.  We already have 
>> sys.set_coroutine_wrapper(), so this isn’t something unprecedented.
> 
> There isn't just one event loop though, and what trampoline to attach a dying 
> coroutine to depends heavily on what event loop it came from.  It seems like 
> a single global value for this in 'sys' would just be ... wrong.

But there can only be one currently running event loop per thread...

Another way is to add sys.set_async_generator_wrapper() (and a getter, so that 
loops can maintain a stack of them).  With it a running event loop can create a 
weak ref to a generator that is created in a coroutine that the loop is 
currently running. And with a weakref it can later finalize the generator.

Yury___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asynchronous cleanup is a problem

[Yury Selivanov]

> On Jul 6, 2016, at 7:06 PM, Nathaniel Smith  wrote:
> 
> On Wed, Jul 6, 2016 at 1:12 PM, Yury Selivanov  wrote:
>> This is an interesting idea, but I wonder if instead of using ‘async with’ 
>> we can actually augment ‘async for’ to do the async finalization.
>> 
>> We can add an __aiter_close__ special method which will return an awaitable. 
>>  In this case, ‘async for’ can always look for that method and call it at 
>> the end of the iteration.  Async generators will implement the method to 
>> make sure that ‘finally’ is always executed (any number of awaits in 
>> ‘finally’ is OK; ‘yield’ expressions cannot be used).
> 
> I was wondering about that too. This is a fairly substantial change to
> how iterators work, though -- currently, it's totally legal and
> sometimes very useful to do things like
> 
> it = open("...")
> # Discard header line (first non-commented line)
> for line in it:
>if not line.startswith("#"):
>break
> for body_line in it:
>…

Right.  __aiter_close__ won’t work.

The problem is that the GC can’t execute async code, and we don’t have any 
control over GC.  What if we add a mechanism to control how async generators 
(AG) are destructed.  Let’s say we add new function to the sys module - 
`sys.set_async_generator_finalizer(finalizer)`.  We already have 
sys.set_coroutine_wrapper(), so this isn’t something unprecedented.

With this new function, when an AG is about to be finalized, the interpreter 
will resurrect it and call the `finalizer`.  The finalizer function will be 
installed by an event loop, and will execute ‘await AG.aclose()’ in the context 
of that loop.

We can issue a ResourceWarning if an AG (not fully exhausted) is GCed and there 
is no async finalizer.

Yury
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asynchronous cleanup is a problem

[Yury Selivanov]

> On Jul 6, 2016, at 3:54 PM, David Beazley  wrote:
> 
> 
>> However, as far as I know curio doesn’t have the ability to schedule an 
>> operation in a synchronous manner by means of something like a Future. Is 
>> that correct? If there is no way in curio to spawn a task to occur later 
>> without having to await on it, then clearly there is no option but to allow 
>> coroutines in __aexit__ and finally: how else could curio operate?
>> 
> 
> Yes, Curio does not utilize Futures or callback functions for that matter.   
> However, I think the real issue at hand might be much more subtle and weird.
> 
> If I understand things correctly, the goal is to implement an asynchronous 
> generator for feeding asynchronous iteration.  This is a new kind of 
> generator that runs inside of a coroutine (so, imagine a coroutine inside of 
> another coroutine).   Without seeing much more, I'd guess it would look 
> something roughly akin to this:
> 
> async def agen():
>   ... some sort of async generator ...
>   async yield value #  Syntax 

I my current WIP branch I just use ‘yield’ inside ‘async def’.  That’s what I 
was going to propose to do in the PEP.

> 
> async def main():
>   async for x in agen():
>  ...
> 
> There's some kind of underlying protocol driving the async iteration, but 
> it's *different* than what is being used by the enclosing coroutines.   Yes, 
> there is a scheduler kernel (or event loop) that makes the outer coroutines 
> run, but that scheduler is not driving the underlying iteration protocol of 
> the async generator part.   So, things get weird when stuff like this happens:
> 
> async def main():
>   async for x in agen():
>   if x == STOP:
> break

Good catch.

[..]

> Since forgetting that last close() step would be easy, naturally an async 
> generator should support the asynchronous context-management protocol.
> 
> async def main():
>   async with agen() as items:
>  async for x in items:
>  if x == STOP:
>break

This is an interesting idea, but I wonder if instead of using ‘async with’ we 
can actually augment ‘async for’ to do the async finalization.

We can add an __aiter_close__ special method which will return an awaitable.  
In this case, ‘async for’ can always look for that method and call it at the 
end of the iteration.  Async generators will implement the method to make sure 
that ‘finally’ is always executed (any number of awaits in ‘finally’ is OK; 
‘yield’ expressions cannot be used).

Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/

Re: [Async-sig] Asynchronous cleanup is a problem

[Yury Selivanov]

> On Jul 5, 2016, at 8:56 PM, Nathaniel Smith  wrote:
[..]
> Starting axiom: async functions / async generators need to be prepared
> for the case where they get garbage collected before being run to
> completion, because, well... this is a thing that can happen.
[..]
> Dismaying conclusion: inside an async function / async generator,
> finally: blocks must never yield (and neither can __aexit__
> callbacks), because they might be invoked during garbage collection.

I agree with David here: coroutines are always running under a scheduler
which, in one way or another, keeps strong references to them.  In curio
it’s a global table of all tasks, in asyncio it’s a chain of references
to callbacks of Tasks/Futures.  The only time a coroutine can be GC’ed 
in asyncio is when the caller did not use ‘await’ on it.

A running coroutine being GC’ed is an exceptional situation, that means 
that there is a bug in your scheduler.  And people actually rely on this
thing.  It’s not so much about __aexit__ returning an awaitable, it’s
about people writing code that awaits in ‘finally’ statements.

That’s why I’m big -1 on changing __aexit__.  If we change __aexit__, 
we should also prohibit awaiting in finally blocks, which is not an 
option.


> For async functions this is... arguably a problem but not super
> urgent, because async functions rarely get garbage collected without
> being run to completion. For async generators it's a much bigger
> problem.

I’m not sure I understand why it’d a problem for async generators.
Since coroutines shouldn’t ever be GC’ed while running, async generators
generally won’t be GC’ed while running too, because they will have
a strong ref from the running coroutine.

I think to makes things simple, we shouldn’t have a ‘close()’
method on async generators at all.  When a running async generator is
GC’ed we’ll make the interpreter to issue a warning.

We might want to add an ‘aclose()’ coroutine method, which will throw 
a GeneratorExit exception (or GeneratorAsyncExit) with the following
semantics:

1. If the running async generator ignores GeneratorAsyncExit and keeps
‘async yielding’, we throw a RuntimeError.

2. If the running async generator receives a GeneratorAsyncExit exception
outside of a try..finally block, the async generator is closed silently.

3. If the running async generator receives a GeneratorAsyncExit exception
inside a ‘finally’ block, it will be able to await on any number of
coroutines inside that block.

Thanks,
Yury

___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/