Re: [Python-Dev] Accepting PEP 492 (async/await)
On 2015-05-05 7:53 PM, Guido van Rossum wrote: Everybody, In order to save myself a major headache I'm hereby accepting PEP 492. I've been following Yury's efforts carefully and I am fully confident that we're doing the right thing here. There is only so much effort we can put into clarifying terminology and explaining coroutines. Somebody should write a tutorial. (I started to write one, but I ran out of time after just describing basic yield.) Thank you, Guido! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
I wonder if you could look at Tkinter for a very different view of the world. While there are ways to integrate socket I/O with the Tcl/Tk event loop, the typical Python app using Tkinter probably moves network I/O (if it has any) to a separate thread, and ignores the delays of disk-based I/O (because modern disk I/O is usually faster than the minimal event response time -- assuming you don't have floppy disks :-). I'm not entirely sure how you would use coroutines with Tkinter, but I could imagine that e.g. mouse tracking code such as found in drawing apps might be written more easily using a loop that uses await (or yield [from]) to get another event rather than as a callback function for the mouse move event. The mechanics of writing a multiplexer that receives Tkinter events and uses them to decide which generator/coroutine to wake up might be too much for your purpose, but it would provide a real-life example of an event loop that's not built for network I/O. On Tue, May 5, 2015 at 3:52 PM, Paul Moore p.f.mo...@gmail.com wrote: On 5 May 2015 at 23:28, Guido van Rossum gu...@python.org wrote: At this point, *all* I'm thinking of is a toy. So, an implementation somewhat parallel to asyncio, but where the event loop just passes control to the next task - so no IO multiplexing. Essentially Greg Ewing's example up to, but not including, Waiting for External Events. And ideally I'd like to think that Waiting for Resources can be omitted in favour of reusing https://docs.python.org/3/library/asyncio-sync.html and https://docs.python.org/3/library/asyncio-queue.html. My fear is, however, that those parts of asyncio aren't reusable for other event loops, and every event loop implementation has to reinvent those wheels. It was never a goal of asyncio to have parts that were directly reusable by other event loops without pulling in (almost) all of asyncio. The interoperability offered by asyncio allows other event loops to implement the same low-level interface as asyncio, or to build on top of asyncio. (This is why the event loop uses callbacks and isn't coroutines/generators all the way down.) Note that asyncio.get_event_loop() may return a loop implemented by some other framework, and the rest of asyncio will then use that event loop. This is enabled by the EventLoopPolicy interface. OK, that's an entirely fair comment. It's difficult to tell from the docs - there's nothing obviously io-related about the task abstraction, or the synchronisation or queue primitives. But there's equally no reason to assume that they would work with another implementation. As I mentioned somewhere else, maybe refactoring the bits of asyncio that can be reused into an asynclib module would be useful. But based on what you said, there's no reason to assume that would be an easy job. And without another event loop implementation, it's not obvious that there's a justification for doing so. What do you hope to learn or teach by creating this toy example? And how do you define a complete event loop? Well, one thing I hope to learn, I guess, is what a complete event loop consists of :-) More broadly, I'd like to get a better feel for what methods are fundamental to an event loop. IIRC, we had this discussion way back at the beginning of the asyncio development when I was unclear about why create_connection had to be an event loop method. In the asyncio context, it has to be because the event loop needs to know when connections get created (excuse me probably misremembering the exact reason from back then). But conversely, it's easy to imagine an event loop unrelated to socket IO that doesn't have a create_connection method. On the other hand, an event loop with no call_soon method seems unlikely. So in essence I'm thinking about what a sensible minimum event loop might be. An event loop ABC, if you like. And following on from there, what useful abstractions (tasks, synchronisation and queue primitives) can be built on top of such a minimal interface. Basically, that's what I'm hoping to learn - what is fundamental (or at least generally applicable) and what is related to the purpose of a given implementation. I've probably got enough from this discussion to try writing up some code and see where it leads me. Paul PS You mentioned that a the callback-based nature of the asyncio event loop is to simplify interoperability with callback-based frameworks like Twisted. I guess the above ignores the possibility of event loops that *aren't* callback-based. Or maybe it doesn't - that's possibly another class of methods (callback-focused ones) that maybe can be separated into their own ABC. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 5 May 2015 at 23:28, Guido van Rossum gu...@python.org wrote: At this point, *all* I'm thinking of is a toy. So, an implementation somewhat parallel to asyncio, but where the event loop just passes control to the next task - so no IO multiplexing. Essentially Greg Ewing's example up to, but not including, Waiting for External Events. And ideally I'd like to think that Waiting for Resources can be omitted in favour of reusing https://docs.python.org/3/library/asyncio-sync.html and https://docs.python.org/3/library/asyncio-queue.html. My fear is, however, that those parts of asyncio aren't reusable for other event loops, and every event loop implementation has to reinvent those wheels. It was never a goal of asyncio to have parts that were directly reusable by other event loops without pulling in (almost) all of asyncio. The interoperability offered by asyncio allows other event loops to implement the same low-level interface as asyncio, or to build on top of asyncio. (This is why the event loop uses callbacks and isn't coroutines/generators all the way down.) Note that asyncio.get_event_loop() may return a loop implemented by some other framework, and the rest of asyncio will then use that event loop. This is enabled by the EventLoopPolicy interface. OK, that's an entirely fair comment. It's difficult to tell from the docs - there's nothing obviously io-related about the task abstraction, or the synchronisation or queue primitives. But there's equally no reason to assume that they would work with another implementation. As I mentioned somewhere else, maybe refactoring the bits of asyncio that can be reused into an asynclib module would be useful. But based on what you said, there's no reason to assume that would be an easy job. And without another event loop implementation, it's not obvious that there's a justification for doing so. What do you hope to learn or teach by creating this toy example? And how do you define a complete event loop? Well, one thing I hope to learn, I guess, is what a complete event loop consists of :-) More broadly, I'd like to get a better feel for what methods are fundamental to an event loop. IIRC, we had this discussion way back at the beginning of the asyncio development when I was unclear about why create_connection had to be an event loop method. In the asyncio context, it has to be because the event loop needs to know when connections get created (excuse me probably misremembering the exact reason from back then). But conversely, it's easy to imagine an event loop unrelated to socket IO that doesn't have a create_connection method. On the other hand, an event loop with no call_soon method seems unlikely. So in essence I'm thinking about what a sensible minimum event loop might be. An event loop ABC, if you like. And following on from there, what useful abstractions (tasks, synchronisation and queue primitives) can be built on top of such a minimal interface. Basically, that's what I'm hoping to learn - what is fundamental (or at least generally applicable) and what is related to the purpose of a given implementation. I've probably got enough from this discussion to try writing up some code and see where it leads me. Paul PS You mentioned that a the callback-based nature of the asyncio event loop is to simplify interoperability with callback-based frameworks like Twisted. I guess the above ignores the possibility of event loops that *aren't* callback-based. Or maybe it doesn't - that's possibly another class of methods (callback-focused ones) that maybe can be separated into their own ABC. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 5/5/2015 6:25 PM, Yury Selivanov wrote: Yes, there is no other popular event loop for 3.4 other than asyncio, There is the tk(inter) event loop which also ships with CPython, and which is commonly used. that uses coroutines based on generators Oh ;-) Tkinter event loop is callback based. AFAIK, so is the asyncio event loop, but that is somehow masked by tasks that interface to coroutines. Do you think the 'somehow' could be adapted to work with the tkinter loop? What I do not understand is how io events become event loop Event instances. For tk, keyboard and mouse actions seen by the OS become tk Events associated with a widget. Some widgets generate events. User code can also generate (pseudo)events. My specific use case is to be able to run a program in a separate process, but display the output in the gui process -- something like this (in Idle, for instance). (Apologies if this misuses the new keywords.) async def menu_handler() ow = OutputWindow(args) # tk Widget proc = subprocess.Popen (or multiprocessing equivalent) out = (stdout from process) await for line in out: ow.write(line) finish() I want the handler to not block event processing, and disappear after finishing. Might 492 make this possible someday? Or would having 'line in pipe' or just 'data in pipe' translated to a tk event likely require a patch to tk? -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
For this you should probably use an integration of asyncio (which can do async subprocess output nicely) with Tkinter. Over in tulip-land there is an demo of such an integration. On Tue, May 5, 2015 at 6:03 PM, Terry Reedy tjre...@udel.edu wrote: On 5/5/2015 6:25 PM, Yury Selivanov wrote: Yes, there is no other popular event loop for 3.4 other than asyncio, There is the tk(inter) event loop which also ships with CPython, and which is commonly used. that uses coroutines based on generators Oh ;-) Tkinter event loop is callback based. AFAIK, so is the asyncio event loop, but that is somehow masked by tasks that interface to coroutines. Do you think the 'somehow' could be adapted to work with the tkinter loop? What I do not understand is how io events become event loop Event instances. For tk, keyboard and mouse actions seen by the OS become tk Events associated with a widget. Some widgets generate events. User code can also generate (pseudo)events. My specific use case is to be able to run a program in a separate process, but display the output in the gui process -- something like this (in Idle, for instance). (Apologies if this misuses the new keywords.) async def menu_handler() ow = OutputWindow(args) # tk Widget proc = subprocess.Popen (or multiprocessing equivalent) out = (stdout from process) await for line in out: ow.write(line) finish() I want the handler to not block event processing, and disappear after finishing. Might 492 make this possible someday? Or would having 'line in pipe' or just 'data in pipe' translated to a tk event likely require a patch to tk? -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Accepting PEP 492 (async/await)
Everybody, In order to save myself a major headache I'm hereby accepting PEP 492. I've been following Yury's efforts carefully and I am fully confident that we're doing the right thing here. There is only so much effort we can put into clarifying terminology and explaining coroutines. Somebody should write a tutorial. (I started to write one, but I ran out of time after just describing basic yield.) I've given Yury clear instructions to focus on how to proceed -- he's to work with another core dev on getting the implementation ready in time for beta 1 (scheduled for May 24, but I think the target date should be May 19). The acceptance is provisional in the PEP 411 sense (stretching its meaning to apply to language changes). That is, we reserve the right to change the specification (or even withdraw it, in a worst-case scenario) until 3.6, although I expect we won't need to do this except for some peripheral issues (e.g. the backward compatibility flags). I now plan to go back to PEP 484 (type hints). Fortunately in that case there's not much *implementation* that will land (just the typing.py module), but there's still a lot of language in the PEP that needs updating (check the PEP 484 tracker https://github.com/ambv/typehinting/issues). -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 492 (async/await)
Congratulations, Yury! -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 492 (async/await)
I totally forgot to publicly congratulate Yury on this PEP. He's put a huge effort into writing the PEP and the implementation and managing the discussion, first on python-ideas, later on python-dev. Congrats, Yury! And thanks for your efforts. Godspeed. On Tue, May 5, 2015 at 4:53 PM, Guido van Rossum gu...@python.org wrote: Everybody, In order to save myself a major headache I'm hereby accepting PEP 492. I've been following Yury's efforts carefully and I am fully confident that we're doing the right thing here. There is only so much effort we can put into clarifying terminology and explaining coroutines. Somebody should write a tutorial. (I started to write one, but I ran out of time after just describing basic yield.) I've given Yury clear instructions to focus on how to proceed -- he's to work with another core dev on getting the implementation ready in time for beta 1 (scheduled for May 24, but I think the target date should be May 19). The acceptance is provisional in the PEP 411 sense (stretching its meaning to apply to language changes). That is, we reserve the right to change the specification (or even withdraw it, in a worst-case scenario) until 3.6, although I expect we won't need to do this except for some peripheral issues (e.g. the backward compatibility flags). I now plan to go back to PEP 484 (type hints). Fortunately in that case there's not much *implementation* that will land (just the typing.py module), but there's still a lot of language in the PEP that needs updating (check the PEP 484 tracker https://github.com/ambv/typehinting/issues). -- --Guido van Rossum (python.org/~guido) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 492 (async/await)
Hi, 2015-05-06 1:53 GMT+02:00 Guido van Rossum gu...@python.org: In order to save myself a major headache I'm hereby accepting PEP 492. Great! Congrats Yury. I've given Yury clear instructions to focus on how to proceed -- he's to work with another core dev on getting the implementation ready in time for beta 1 (scheduled for May 24, but I think the target date should be May 19). The implementation takes place at: https://bugs.python.org/issue24017 Yury works at https://github.com/1st1/cpython in git branches. I already sent a first review. But I don't feel able to review the change on the grammar, parser or things like that. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: What is the real goal?
On Fri May 1 23:58:26 CEST 2015, Yury Selivanov wrote: Yes, you can't use 'yield from' in __exit__/__enter__ in current Python. What do you mean by can't use? It probably executed without errors, but it didn't run the generators. True. But it did return the one created by __enter__, so it could be bound to a variable and iterated within the block. There isn't an easy way to run the generator created by __exit__, and I'm not coming up with any obvious scenarios where it would be a sensible thing to do (other than using with on a context manager that *does* return a future instead of finishing). That said, I'm still not seeing why the distinction is so important that we have to enforce it at a language level, as opposed to letting the framework do its own enforcement. (And if the reason is performance, then make the checks something that can be turned off, or offer a fully instrumented loop as an alternative for debugging.) Is the intent to do anything more than preface execution with: import asynchio.coroutines asynchio.coroutines._DEBUG = True If you enable it after you import asyncio, then asyncio itself won't be instrumented. Why does asynchio itself have to wrapped? Is that really something normal developers need to debug, or is it only for developing the stdlib itself? Yes, normal developers need asyncio to be instrumented, otherwise you won't know what you did wrong when you called some asyncio code without 'await' for example. I'll trust you that it *does* work that way, but this sure sounds to me as though the framework isn't ready to be frozen with syntax, and maybe not even ready for non-provisional stdlib inclusion. I understand that the disconnected nature of asynchronous tasks makes them harder to debug. I heartily agree that the event loop should offer some sort of debug facility to track this. But the event loop is supposed to be pluggable. Saying that this requires not merely a replacement, or even a replacement before events are added, but a replacement made before python ever even loads the default version ... That seems to be much stronger than sys.settrace -- more like instrumenting the ceval loop itself. And that is something that ordinary developers shouldn't have to do. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
tds333 at gmail.com tds333 at gmail.com writes: Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. [...] We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. [...] Hi Wolfgang, You may want to see what I just posted on python-ideas. What I wrote about is related to several things you mention, and might provide a remedy. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
Paul, On 2015-05-05 5:54 PM, Paul Moore wrote: On 5 May 2015 at 22:38, Yury Selivanov yselivanov...@gmail.com wrote: n 2015-05-05 5:01 PM, Paul Moore wrote: On 5 May 2015 at 21:00, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. Given that the stdlib doesn't provide an example of such a UI loop, what would a 3rd party module need to implement to provide such a thing? Can any of the non-IO related parts of asyncio be reused for the purpose, or must the 3rd party module implement everything from scratch? The idea is that you integrate processing of UI events to your event loop of choice. For instance, Twisted has integration for QT and other libraries [1]. This way you can easily combine async network (or OS) calls with your UI logic to avoid callback hell. We seem to be talking at cross purposes. You say the PEP is *not* exclusively intended for use with asyncio. You mention UI loops, but when asked how to implement such a loop, you say that I integrate UI events into my event loop of choice. But what options do I have for my event loop of choice? Please provide a concrete example that isn't asyncio. Yes, there is no other popular event loop for 3.4 other than asyncio, that uses coroutines based on generators (as far as I know). And yes, the PEP is not exclusively intended for use with asyncio, but asyncio is the only library that ships with Python, and is Python 3 ready, so its users will be the first ones to directly benefit from this proposal. Can I use PEP 492 with Twisted (I doubt it, as Twisted doesn't use yield from, which is Python 3.x only)? I contend that there *is* no concrete example that currently exists, so I'm asking what I'd need to do to write one. You pointed at qamash, but that seems to be subclassing asyncio, so isn't something that isn't asyncio. When Twisted is ported to Python 3, I'd be really surprised if it doesn't allow to use the new syntax. @inlineCallbacks implements a trampoline to make 'yields' work. This is a much slower approach than using 'yield from' (and 'await' from PEP 492). Not mentioning 'async with' and 'async for' features. (There shouldn't be a problem to support both @inlineCallbacks and PEP 492 approach, if I'm not missing something). Note that I don't have a problem with there being no existing implementation other than asyncio. I'd just like it if we could be clear over exactly what we mean when we say the PEP is not tied to asyncio. Well, the PEP is not tied to asyncio -- this is correct. *The new syntax and new protocols know nothing about asyncio*. asyncio will know about the PEP by implementing new protocols where required etc (but supporting these new features isn't in the scope of the PEP). It feels like the truth currently is you can write your own async framework that uses the new features introduced by the PEP. I fully expect that *if* there's a need for async frameworks that aren't fundamentally IO multiplexors, then it'll get easier to write them over time (the main problem right now is a lack of good tutorial examples of how to do so). But at the moment, asyncio seems to be the only game in town (and I can imagine that it'll always be the main IO multiplexor, unless existing frameworks like Twisted choose to compete rather than integrate). Agree. But if the existing frameworks choose to compete, or someone decides to write something better than asyncio, they can benefit from PEP 492. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On Tue, May 5, 2015 at 2:40 PM, Paul Moore p.f.mo...@gmail.com wrote: On 5 May 2015 at 22:25, Guido van Rossum gu...@python.org wrote: [Paul:] I'd be interested in writing, for instructional purposes, a toy but complete event loop. But I'm *not* really interested in trying to reverse engineer the required interface. This is a great idea. What kind of application do you have in mind? At this point, *all* I'm thinking of is a toy. So, an implementation somewhat parallel to asyncio, but where the event loop just passes control to the next task - so no IO multiplexing. Essentially Greg Ewing's example up to, but not including, Waiting for External Events. And ideally I'd like to think that Waiting for Resources can be omitted in favour of reusing https://docs.python.org/3/library/asyncio-sync.html and https://docs.python.org/3/library/asyncio-queue.html. My fear is, however, that those parts of asyncio aren't reusable for other event loops, and every event loop implementation has to reinvent those wheels. It was never a goal of asyncio to have parts that were directly reusable by other event loops without pulling in (almost) all of asyncio. The interoperability offered by asyncio allows other event loops to implement the same low-level interface as asyncio, or to build on top of asyncio. (This is why the event loop uses callbacks and isn't coroutines/generators all the way down.) Note that asyncio.get_event_loop() may return a loop implemented by some other framework, and the rest of asyncio will then use that event loop. This is enabled by the EventLoopPolicy interface. When I say the required interface I'm thinking in terms of what's needed to allow reuse of the generic parts of asyncio. If nothing of asyncio is generic in those terms, then the exercise will be futile (except in the negative sense of confirming that there are no reusable async components in the stdlib). Paul What do you hope to learn or teach by creating this toy example? And how do you define a complete event loop? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: Please mention the Event Loop
Jim, On 2015-05-05 5:09 PM, Jim J. Jewett wrote: On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote: It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly). [..] Proposed second paragraph of the abstract: This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses yield as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed. Thank you for this suggestion. I've added it to the PEP: https://hg.python.org/peps/rev/7ac132b24f1f Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 5 May 2015 at 23:25, Yury Selivanov yselivanov...@gmail.com wrote: Note that I don't have a problem with there being no existing implementation other than asyncio. I'd just like it if we could be clear over exactly what we mean when we say the PEP is not tied to asyncio. Well, the PEP is not tied to asyncio -- this is correct. *The new syntax and new protocols know nothing about asyncio*. asyncio will know about the PEP by implementing new protocols where required etc (but supporting these new features isn't in the scope of the PEP). Thanks. That's something that may be worth explicitly noting in the PEP (I don't recall it from when I last looked but that was a while ago). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On May 5, 2015 2:14 PM, Guido van Rossum gu...@python.org wrote: In the PEP 492 world, these concepts map as follows: - Future translates to something with an __await__ method (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - asyncio coroutine maps to PEP 492 coroutine object (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - either of the above maps to awaitable. Err, aren't the first and third definitions above identical? Surely we want to say: an async def function is a convenient shorthand for creating a custom awaitable (exactly like how generators are a convenient shorthand for creating custom iterators), and a Future is-an awaitable that also adds some extra methods. -n ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 3:01 PM, Nathaniel Smith n...@pobox.com wrote: On May 5, 2015 2:14 PM, Guido van Rossum gu...@python.org wrote: In the PEP 492 world, these concepts map as follows: - Future translates to something with an __await__ method (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - asyncio coroutine maps to PEP 492 coroutine object (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - either of the above maps to awaitable. Err, aren't the first and third definitions above identical? Surely we want to say: an async def function is a convenient shorthand for creating a custom awaitable (exactly like how generators are a convenient shorthand for creating custom iterators), and a Future is-an awaitable that also adds some extra methods. The current PEP 492 proposal does endow the object returned by calling an async function (let's call it a coroutine object) with an __await__ method. And there's a good reason for this -- the bytecode generated for await treats coroutine objects special, just like the bytecode generated for yield-from treats generator objects special. The special behavior they have in common is the presence of send() and throw() methods, which are used to allow send() and throw() calls on the outer generator to be passed into the inner generator with minimal fuss. (This is the reason why yield from X is *not* equivalent to for x in X: yield x.) @Yury: I have a feeling the PEP could use more clarity here -- perhaps the section Await Expression should explain what the interepreter does for each type of awaitable? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: async/await in Python; version 5
Hi python-dev, Updated version of the PEP is below. Quick summary of changes: 1. set_coroutine_wrapper and get_coroutine_wrapper functions are now thread-specific (like settrace etc). 2. Updated Abstract Rationale sections. 3. RuntimeWarning is always raised when a coroutine wasn't awaited on. This is in addition to what 'set_coroutine_wrapper' will/can do. 4. asyncio.async is renamed to asyncio.ensure_future; it will be deprecated in 3.5. 5. Uses of async/await in CPython codebase are documented. 6. Other small edits and updates. Thanks, Yury PEP: 492 Title: Coroutines with async and await syntax Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov yseliva...@sprymix.com Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Apr-2015 Python-Version: 3.5 Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015, 05-May-2015 Abstract The growth of Internet and general connectivity has triggered the proportionate need for responsive and scalable code. This proposal aims to answer that need by making writing explicitly asynchronous, concurrent Python code easier and more Pythonic. It is proposed to make *coroutines* a proper standalone concept in Python, and introduce new supporting syntax. The ultimate goal is to help establish a common, easily approachable, mental model of asynchronous programming in Python and make it as close to synchronous programming as possible. We believe that the changes proposed here will help keep Python relevant and competitive in a quickly growing area of asynchronous programming, as many other languages have adopted, or are planning to adopt, similar features: [2]_, [5]_, [6]_, [7]_, [8]_, [10]_. Rationale and Goals === Current Python supports implementing coroutines via generators (PEP 342), further enhanced by the ``yield from`` syntax introduced in PEP 380. This approach has a number of shortcomings: * It is easy to confuse coroutines with regular generators, since they share the same syntax; this is especially true for new developers. * Whether or not a function is a coroutine is determined by a presence of ``yield`` or ``yield from`` statements in its *body*, which can lead to unobvious errors when such statements appear in or disappear from function body during refactoring. * Support for asynchronous calls is limited to expressions where ``yield`` is allowed syntactically, limiting the usefulness of syntactic features, such as ``with`` and ``for`` statements. This proposal makes coroutines a native Python language feature, and clearly separates them from generators. This removes generator/coroutine ambiguity, and makes it possible to reliably define coroutines without reliance on a specific library. This also enables linters and IDEs to improve static code analysis and refactoring. Native coroutines and the associated new syntax features make it possible to define context manager and iteration protocols in asynchronous terms. As shown later in this proposal, the new ``async with`` statement lets Python programs perform asynchronous calls when entering and exiting a runtime context, and the new ``async for`` statement makes it possible to perform asynchronous calls in iterators. Specification = This proposal introduces new syntax and semantics to enhance coroutine support in Python. This specification presumes knowledge of the implementation of coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax changes proposed here comes from the asyncio framework (PEP 3156) and the Cofunctions proposal (PEP 3152, now rejected in favor of this specification). From this point in this document we use the word *native coroutine* to refer to functions declared using the new syntax. *generator-based coroutine* is used where necessary to refer to coroutines that are based on generator syntax. *coroutine* is used in contexts where both definitions are applicable. New Coroutine Declaration Syntax The following new syntax is used to declare a *native coroutine*:: async def read_data(db): pass Key properties of *coroutines*: * ``async def`` functions are always coroutines, even if they do not contain ``await`` expressions. * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function. * Internally, two new code object flags were introduced: - ``CO_COROUTINE`` is used to enable runtime detection of *coroutines* (and migrating existing code). - ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines* (defined with new syntax.) All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and ``CO_GENERATOR`` flags set. * Regular generators, when called, return a *generator object*; similarly, coroutines return a *coroutine object*. * ``StopIteration`` exceptions are not propagated out of coroutines, and are replaced with a
Re: [Python-Dev] PEP 492: What is the real goal?
On 30 April 2015 at 09:50, Arnaud Delobelle arno...@gmail.com wrote: I'm flexible about how we name 'async def' functions. I like to call them coroutines, because that's what they are, and that's how asyncio calls them. It's also convenient to use 'coroutine-object' to explain what is the result of calling a coroutine. I'd like the object created by an 'async def' statement to be called a 'coroutine function' and the result of calling it to be called a 'coroutine'. That would be an improvement over the confusing terminology in the PEP atm. The PEP proposes to name the inspect functions inspect.iscoroutine() and inspect.iscoroutinefunction(). According to the PEP iscoroutine() identifies coroutine objects and iscoroutinefunction() identifies coroutine functions -- a term which is not defined in the PEP but presumably means what the PEP calls a coroutine in the glossary. Calling the async def function an async function and the object it returns a coroutine makes for the clearest terminology IMO (provided the word coroutine is not also used for anything else). It would help to prevent both experienced and new users from confusing the two related but necessarily distinct concepts. Clearly distinct terminology makes it easier to explain/discuss something if nothing else because it saves repeating definitions all the time. -- Oscar ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: What is the real goal?
Hi Oscar, I've updated the PEP with some fixes of the terminology: https://hg.python.org/peps/rev/f156b272f860 I still think that 'coroutine functions' and 'coroutines' is a better pair than 'async functions' and 'coroutines'. First, it's similar to existing terminology for generators. Second, it's less confusing. With pep492 at some point, using generators to implement coroutines won't be a wide spread practice, so 'async def' functions will be the only language construct that returns them. Yury On 2015-05-05 12:01 PM, Oscar Benjamin wrote: On 30 April 2015 at 09:50, Arnaud Delobelle arno...@gmail.com wrote: I'm flexible about how we name 'async def' functions. I like to call them coroutines, because that's what they are, and that's how asyncio calls them. It's also convenient to use 'coroutine-object' to explain what is the result of calling a coroutine. I'd like the object created by an 'async def' statement to be called a 'coroutine function' and the result of calling it to be called a 'coroutine'. That would be an improvement over the confusing terminology in the PEP atm. The PEP proposes to name the inspect functions inspect.iscoroutine() and inspect.iscoroutinefunction(). According to the PEP iscoroutine() identifies coroutine objects and iscoroutinefunction() identifies coroutine functions -- a term which is not defined in the PEP but presumably means what the PEP calls a coroutine in the glossary. Calling the async def function an async function and the object it returns a coroutine makes for the clearest terminology IMO (provided the word coroutine is not also used for anything else). It would help to prevent both experienced and new users from confusing the two related but necessarily distinct concepts. Clearly distinct terminology makes it easier to explain/discuss something if nothing else because it saves repeating definitions all the time. -- Oscar ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. (2) The new syntax is intended to make it easier to recognize when a task's execution may be interrupted by arbitrary other tasks, and the interrupted task therefore has to revalidate assumptions about shared data. With threads, CPython can always suspend a task between op-codes, but with a sufficiently comprehensive loop (and sufficiently coooperative tasks), tasks *should* only be suspended when they make an explicit request to *wait* for an answer, and these points *should* be marked syntactically. (3) The new constructs explicitly do NOT support any sort of concurrent execution within a task; they are for use precisely when otherwise parallel subtasks are being linearized by pausing and waiting for the results. Over-simplifications 4-6 assume a world with standardized futures based on concurrent.futures, where .result either returns the result or raises the exception (or raises another exception about timeout or cancellation). [Note that the actual PEP uses iteration over the results of a new __await__ magic method, rather than .result on the object itself. I couldn't tell whether this was for explicit marking, or just for efficiency in avoiding future creation.] (4) await EXPR is just syntactic sugar for EXPR.result except that, by being syntax, it better marks locations where unrelated tasks might have a chance to change shared data. [And that, as currently planned, the result of an await isn't actually the result; it is an iterator of results.] (5) async def is just syntactic sugar for def, except that, by being syntax, it better marks the signatures of functions and methods where unrelated tasks might have a chance to change shared data after execution has already begun. (5A) As the PEP currently stands, it is also a promise that the function will NOT produce a generator used as an iterator; if a generator-iterator needs to wait for something else at some point, that will need to be done differently. I derive this limitation from It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function. but I don't understand how this limitation works with things like a per-line file iterator that might need to wait for the file to be initially opened. (6) async with EXPR as VAR: would be equivalent to: with EXPR as VAR: except that __enter__() would be replaced by next(await __enter__()) # __enter__().result __exit__() would be replaced by next(await __exit__()) # __exit__().result (7) async for elem in iter: would be shorthand for: for elem in iter: elem = next(await elem) # elem.result -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote: On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 7:27 AM, Wolfgang wrote: Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of trust us, it makes sense if you need it, everyone else can ignore it. Watch David Beazley's talk from PyCon this year and you can watch him basically re-implement asyncio on stage in under 45 minutes. It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly). Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. I think this ties the concept of adding syntax to Python to make coroutine-based programming easier too much to asyncio; the latter is just an implementation of the former. This PEP doesn't require asyncio beyond the fact that will be what provides the initial event loop in the stdlib. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get turn the wheel left to make the car go left. Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. I think the goal is make coroutines easier to use and does not directly relate to asyncio. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Yep. Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? I believe so; you need something to implement __await__. This is true in any language that implements co-routines. Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. You forgot to append ... yet to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that? And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). And how much extra will this PEP add? Those may not be fair
Re: [Python-Dev] ABCs - Re: PEP 492: async/await in Python; version 4
On Mon, May 4, 2015 at 10:35 AM, Jim J. Jewett jimjjew...@gmail.com wrote: Which reminds me ... *should* the await keyword work with any future, or is it really intentionally restricted to use with a single library module and 3rd party replacements? You can make any Future type work with await by adding an __await__ method. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 7:27 AM, Wolfgang wrote: Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of trust us, it makes sense if you need it, everyone else can ignore it. Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that? And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively large amount of new syntax and special methods I have to ignore because they don't make sense to me). That's OK, but I think I am being reasonable to ask for some sense of the level of benefits others are getting to balance out the costs I incur. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: What is the real goal?
On Tue, May 5, 2015 at 9:48 AM, Yury Selivanov yselivanov...@gmail.com wrote: Hi Oscar, I've updated the PEP with some fixes of the terminology: https://hg.python.org/peps/rev/f156b272f860 I still think that 'coroutine functions' and 'coroutines' is a better pair than 'async functions' and 'coroutines'. Yes. This subtopic is now closed for further debate (please). -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Paul, On 2015-05-05 3:14 PM, Paul Moore wrote: On 5 May 2015 at 19:25, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 7:27 AM, Wolfgang wrote: Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. But the terminology discussion appears to revolve around people finding the various concepts involved in asyncio (particularly the new PEP, but also to an extent the existing implementation) confusing. I can confirm, having tried to work through the asyncio docs, that the underlying concepts and how they are explained, are confusing to an outsider. I agree. We have to improve asyncio docs in this area. That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of trust us, it makes sense if you need it, everyone else can ignore it. Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. Again, PEP 492 is not only for asyncio. *Any* framework can use it, including Twisted. As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. I can't see any fundamental problem with the PEP behind such discussions. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. There is aiohttp library [1], which provides a client API similar to requests. And if you want to write high performance networking server in python3 you *will* choose asyncio (or gevent/twisted in python2). And PEP 492 is aimed to make this whole async stuff more accessible to an average user. As another example, in Twisted I could use defer_to_thread to integrate Oracle database access into a twisted application (that's what the twisted database stuff did under the hood). Can I do that with asyncio? Will the syntax in the PEP help, hinder or be irrelevant to that? You can use 'loop.run_in_executor' in asyncio. It returns a future that you can await on. You can also provide a nice facade for your Oracle-database code that provides a nice API but uses asyncio thread executor behind the scenes. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Without async versions of requests and similar, how much of a chunk of the networking/web area will asyncio take? (Genuine question, I have no idea). There are some things (like websockets) that are hard to implement correctly in existing frameworks like django and flask. And these kind of things are becoming more and more important. Languages like Go were designed specifically to allow writing efficient And how much extra will this PEP add? Those may not be fair questions (and even if they are fair, the answers are probably unknowable), but as an outsider, I feel only the costs of the asyncio implementation (a new library that I don't understand, and now a relatively
[Python-Dev] PEP 492: async/await in Python; version 4
Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have native coroutines and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with a. Easy to confuse with a for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called async. Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
(Yury gave similar responses, so (a) I'll just respond here, and (b) it's encouraging that you both responded so quickly with the same message) On 5 May 2015 at 20:44, Brett Cannon br...@python.org wrote: That's not to say that everything needs to be beginner-friendly, but it *does* mean that it's hard for the wider Python community to meaningfully comment, or evaluate or sanity-check the design. We're left with a sense of trust us, it makes sense if you need it, everyone else can ignore it. Watch David Beazley's talk from PyCon this year and you can watch him basically re-implement asyncio on stage in under 45 minutes. It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly). I'll watch that - it should be fun. But I have seen things like that before, and I've got an idea how to write an event loop. You're right that it's easy to lose track of the fundamentally simple idea in all the complex discussions. To me that feels like a peculiar failure of the abstraction, in that in some circumstances it makes things feel *more* complex than they are :-) Personally, I feel as if PEP 492 is looking a little premature - maybe the focus should be on making asyncio more accessible first, and *then* adding syntax. I think this ties the concept of adding syntax to Python to make coroutine-based programming easier too much to asyncio; the latter is just an implementation of the former. This PEP doesn't require asyncio beyond the fact that will be what provides the initial event loop in the stdlib. It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. It would probably be helpful to have a concrete example of a basic event loop that did *nothing* but schedule tasks. No IO waiting or similar, just scheduling. I have a gut feeling that event loops are more than just asyncio, but without examples to point to it's hard to keep a focus on that fact. And even harder to isolate what is an event loop mechanism from what is asyncio specific. For example, asyncio.BaseEventLoop has a create_connection method. That's *obviously* not a fundamental aspect of a generic event loop, But call_soon (presumably) is. Having a documented basic event loop interface would probably help emphasise the idea than event loops don't have to be asyncio. (Actually, what *is* the minimal event loop interface that is needed for the various task/future mechanisms to work, independently of asyncio? And what features of an event loop etc are needed for the PEP, if it's being used outside of asyncio?) I guess the other canonical event loop use case is GUI system message dispatchers. You can argue that the syntax is needed to help make async more accessible - but if that's the case then the terminology debates and confusion are clear evidence that it's not succeeding in that goal. Perhaps, but arguing about the nitty-gritty details of something doesn't automatically lead to a clearer understanding of the higher level concept. Discussing how turning a steering wheel in a car might help you grasp how cars turn, but it isn't a requirement to get turn the wheel left to make the car go left. Fair point. If only I could avoid driving into walls :-) Of course, that's based on my perception of one of the goals of the PEP as being make coroutines and asyncio more accessible, If the actual goals are different, my conclusion is invalid. I think the goal is make coroutines easier to use and does not directly relate to asyncio. OK. But in that case, some examples using a non-asyncio toy just schedule tasks event loop might help. Well, twisted always had defer_to_thread. Asyncio has run_in_executor, but that seems to be callback-based rather than coroutine-based? Yep. ... and so you can't use it with async/await? Many people use requests for their web access. There are good reasons for this. Are you saying that until someone steps up and writes an async implementation of requests, I have to make a choice - requests or asyncio? I believe so; you need something to implement __await__. This is true in any language that implements co-routines. Unfortunately, I can't see myself choosing asyncio in that situation. Which again means that asyncio becomes something that the average user can't use. Which in turn further entrenches it as a specialist-only tool. You forgot to append ... yet to that statement. Just because something isn't available out of the box without some effort to support doesn't mean it will never happen, else there would be absolutely no Python 3 users out there. Fair point. Yuri mentioned aiohttp, as well. The one difference between this and Python 2/3, is that here you *have* to have two separate implementations. There's no equivalent of a shared source async and synchronous
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 5 May 2015 at 21:38, Guido van Rossum gu...@python.org wrote:
Jumping in to correct one fact.
On Tue, May 5, 2015 at 12:44 PM, Brett Cannon br...@python.org wrote:
On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote:
Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
but that seems to be callback-based rather than coroutine-based?
Yep.
The run_in_executor call is not callback-based -- the confusion probably
stems from the name of the function argument ('callback'). It actually
returns a Future representing the result (or error) of an operation, where
the operation is represented by the function argument. So if you have e.g. a
function
def factorial(n):
return 1 if n = 0 else n*factorial(n-1)
you can run it in an executor from your async(io) code like this:
loop = asyncio.get_event_loop()
result = yield from loop.run_in_executor(factorial, 100)
(In a PEP 492 coroutine substitute await for yield from.)
Thanks, that's an important correction. Given that, run_in_executor is
the link to blocking calls that I was searching for. And yes, the
callback terminology does make this far from obvious, unfortunately.
As does the point at which it's introduced (before futures have been
described) and the fact that it says this method is a coroutine
rather than this method returns a Future[1].
Paul
[1] I'm still struggling to understand the terminology, so if those
two statements are equivalent, that's not yet obvious to me.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
One small clarification: On Tue, May 5, 2015 at 12:40 PM, Jim J. Jewett jimjjew...@gmail.com wrote: [...] but I don't understand how this limitation works with things like a per-line file iterator that might need to wait for the file to be initially opened. Note that PEP 492 makes it syntactically impossible to use a coroutine function to implement an iterator using yield; this is because the generator machinery is needed to implement the coroutine machinery. However, the PEP allows the creation of asynchronous iterators using classes that implement __aiter__ and __anext__. Any blocking you need to do can happen in either of those. You just use `async for` to iterate over such an asynchronous stream. (There's an issue with actually implementing an asynchronous stream mapped to a disk file, because I/O multiplexing primitives like select() don't actually support waiting for disk files -- but this is an unrelated problem, and asynchronous streams are useful to handle I/O to/from network connections, subprocesses (pipes) or local RPC connections. Checkout the streams https://docs.python.org/3/library/asyncio-stream.html and subprocess https://docs.python.org/3/library/asyncio-subprocess.html submodules of the asyncio package. These streams would be great candidates for adding __aiter__/__anext__ to support async for-loops, so the idiom for iterating over them can once again closely resemble the idiom for iterating over regular (synchronous) streams using for-loops.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. (2) The new syntax is intended to make it easier to recognize when a task's execution may be interrupted by arbitrary other tasks, and the interrupted task therefore has to revalidate assumptions about shared data. With threads, CPython can always suspend a task between op-codes, but with a sufficiently comprehensive loop (and sufficiently coooperative tasks), tasks *should* only be suspended when they make an explicit request to *wait* for an answer, and these points *should* be marked syntactically. (3) The new constructs explicitly do NOT support any sort of concurrent execution within a task; they are for use precisely when otherwise parallel subtasks are being linearized by pausing and waiting for the results. Yes. Over-simplifications 4-6 assume a world with standardized futures based on concurrent.futures, where .result either returns the result or raises the exception (or raises another exception about timeout or cancellation). [Note that the actual PEP uses iteration over the results of a new __await__ magic method, rather than .result on the object itself. I couldn't tell whether this was for explicit marking, or just for efficiency in avoiding future creation.] (4) await EXPR is just syntactic sugar for EXPR.result except that, by being syntax, it better marks locations where unrelated tasks might have a chance to change shared data. [And that, as currently planned, the result of an await isn't actually the result; it is an iterator of results.] I'm not sure how to comment on (4). Perhaps I don't understand some notation that you're using. If anything, it's more of a syntactic sugar for 'yield from EXPR'. (5) async def is just syntactic sugar for def, except that, by being syntax, it better marks the signatures of functions and methods where unrelated tasks might have a chance to change shared data after execution has already begun. It also sets CO_COROUTINE | CO_GENERATOR flags, that are very important. (5A) As the PEP currently stands, it is also a promise that the function will NOT produce a generator used as an iterator; if a generator-iterator needs to wait for something else at some point, that will need to be done differently. I derive this limitation from It is a ``SyntaxError`` to have ``yield`` or ``yield from`` expressions in an ``async`` function. but I don't understand how this limitation works with things like a per-line file iterator that might need to wait for the file to be initially opened. Per-line file iterator can be implemented with __aiter__, __anext__ protocol. __aiter__ is a coroutine, you can open/start reading your file there. (6) async with EXPR as VAR: would be equivalent to: with EXPR as VAR: except that __enter__() would be replaced by next(await __enter__()) # __enter__().result __exit__() would be replaced by next(await __exit__()) # __exit__().result I'm not sure I understand what you mean by next(await EXPR) notation. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On May 5, 2015 12:40 PM, Jim J. Jewett jimjjew...@gmail.com wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? [...snip...] [Note that the actual PEP uses iteration over the results of a new __await__ magic method, rather than .result on the object itself. I couldn't tell whether this was for explicit marking, or just for efficiency in avoiding future creation.] (4) await EXPR is just syntactic sugar for EXPR.result except that, by being syntax, it better marks locations where unrelated tasks might have a chance to change shared data. [And that, as currently planned, the result of an await isn't actually the result; it is an iterator of results.] This is where you're missing a key idea. (And I agree that more high-level docs are very much needed!) Remember that this is just regular single threaded python code, so just writing EXPR.result cannot possibly cause the current task to pause and another one to start running, and then magically switch back somehow when the result does become available. Imagine trying to implement a .result attribute that does that -- it's impossible. Writing 'x = await socket1.read(1)' is actually equivalent to writing a little loop like: while True: # figure out what we need to happen to make progress needed = data from socket 1 # suspend this function, # and send the main loop a message telling it what we need reply = (yield needed) # okay, the main loop woke us up again # let's see if they've sent us back what we asked for if reply.type == data from socket 1: # got it! x = reply.payload break else: # if at first you don't succeed... continue (Now stare at the formal definition of 'yield from' until you see how it maps onto the above... And if you're wondering why we need a loop, think about the case where instead of calling socket.read we're calling http.get or something that requires multiple steps to complete.) So there actually is semantically no iterator here -- the thing that looks like an iterator is actually the chatter back and forth between the lower-level code and the main loop that is orchestrating everything. Then when that's done, it returns the single result. -n ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Jumping in to correct one fact.
On Tue, May 5, 2015 at 12:44 PM, Brett Cannon br...@python.org wrote:
On Tue, May 5, 2015 at 3:14 PM Paul Moore p.f.mo...@gmail.com wrote:
Well, twisted always had defer_to_thread. Asyncio has run_in_executor,
but that seems to be callback-based rather than coroutine-based?
Yep.
The run_in_executor call is not callback-based -- the confusion probably
stems from the name of the function argument ('callback'). It actually
returns a Future representing the result (or error) of an operation, where
the operation is represented by the function argument. So if you have e.g.
a function
def factorial(n):
return 1 if n = 0 else n*factorial(n-1)
you can run it in an executor from your async(io) code like this:
loop = asyncio.get_event_loop()
result = yield from loop.run_in_executor(factorial, 100)
(In a PEP 492 coroutine substitute await for yield from.)
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 1:39 PM, Paul Moore p.f.mo...@gmail.com wrote: It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-05 4:39 PM, Paul Moore wrote: Is there anyone who feels they could write a stripped down but working example of a valid Python event loop*without* the asyncio aspects? Or is that what David Beazley's talk does? Yes, in David's talk, where he starts to use 'yield from' you can simply use new coroutines. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Hi Wolfgang, On 2015-05-05 7:27 AM, Wolfgang wrote: Hi, [..] Even the discussion on python-dev suggests there is some time needed to finalize all this. I'd say that: 80% of the recent discussion of the PEP is about terminology. 10% is about whether we should have __future__ import or not. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. asyncio and twisted answered these questions ;) The answer is that you have to write async implementations. gevent has a different answer, but greenlents/stackless is something that will never be merged in CPython and other implementations. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. networking stuff, and in particular, web, is a huge part of current Python usage. Please don't underestimate that. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 1:44 PM, Paul Moore p.f.mo...@gmail.com wrote:
[Guido]
The run_in_executor call is not callback-based -- the confusion probably
stems from the name of the function argument ('callback'). It actually
returns a Future representing the result (or error) of an operation,
where
the operation is represented by the function argument. So if you have
e.g. a
function
def factorial(n):
return 1 if n = 0 else n*factorial(n-1)
you can run it in an executor from your async(io) code like this:
loop = asyncio.get_event_loop()
result = yield from loop.run_in_executor(factorial, 100)
(In a PEP 492 coroutine substitute await for yield from.)
Thanks, that's an important correction. Given that, run_in_executor is
the link to blocking calls that I was searching for. And yes, the
callback terminology does make this far from obvious, unfortunately.
As does the point at which it's introduced (before futures have been
described) and the fact that it says this method is a coroutine
rather than this method returns a Future[1].
Paul
[1] I'm still struggling to understand the terminology, so if those
two statements are equivalent, that's not yet obvious to me.
I apologize for the confusing documentation. We need more help from
qualified tech writers! Writing PEP 3156 was a huge undertaking for me;
after that I was exhausted and did not want to take on writing the end user
documentation as well, so it was left unfinished. :-(
In PEP 3156 (asyncio package) there are really three separate concepts:
- Future, which is a specific class (of which Task is a subclass);
- coroutine, by which in this context is meant a generator object obtained
by calling a generator function decorated with @asyncio.coroutine and
written to conform to the asyncio protocol for coroutines (i.e. don't use
bare yield, only use yield from, and the latter always with either a Future
or a coroutine as argument);
- either of the above, which is actually the most common requirement --
most asyncio functions that support one also support the other, and either
is allowable as the argument to `yield from`.
In the implementation we so often flipped between Future and coroutine that
I imagine sometimes the implementation and docs differ; also, we don't have
a good short name for either of the above so we end up using one or the
other as a shorthand.
*Unless* you want to attach callbacks, inspect the result or exception, or
cancel it (all of which require a Future), your code shouldn't be concerned
about the difference -- you should just use `res = yield from func(args)`
and use try/except to catch exceptions if you care. And if you do need a
Future, you can call the function asyncio.async() on it (which in PEP 492
is renamed to ensure_future()).
In the PEP 492 world, these concepts map as follows:
- Future translates to something with an __await__ method (and asyncio
Futures are trivially made compliant by defining Future.__await__ as an
alias for Future.__iter__);
- asyncio coroutine maps to PEP 492 coroutine object (either defined
with `async def` or a generator decorated with @types.coroutine -- note
that @asyncio.coroutine incorporates the latter);
- either of the above maps to awaitable.
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: Please mention the Event Loop
On 5 May 2015 at 22:09, Jim J. Jewett jimjjew...@gmail.com wrote: Proposed second paragraph of the abstract: This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses yield as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed. +1. If that's not accurate then by all means correct any mistakes in it. But assuming it *is* accurate, it would help a lot. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 2015-05-05 5:31 PM, Jim J. Jewett wrote: Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote: As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. Not just How?, but Why?. Why do they*need* to be disambiguated? To clearly show how one interacts with the other, to explain how backwards compatibility is implemented, and to better illustrate some additional (and necessary) restrictions we put on 'async def' coroutines. Otherwise, the PEP would be completely unreadable :) Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On Tue, May 5, 2015 at 2:29 PM, Paul Moore p.f.mo...@gmail.com wrote: On 5 May 2015 at 22:12, Guido van Rossum gu...@python.org wrote: I apologize for the confusing documentation. We need more help from qualified tech writers! Writing PEP 3156 was a huge undertaking for me; after that I was exhausted and did not want to take on writing the end user documentation as well, so it was left unfinished. :-( Fair enough. When I properly document one of my projects, *then* I'll think about complaining :-) These things happen. In PEP 3156 (asyncio package) there are really three separate concepts: - Future, which is a specific class (of which Task is a subclass); - coroutine, by which in this context is meant a generator object obtained by calling a generator function decorated with @asyncio.coroutine and written to conform to the asyncio protocol for coroutines (i.e. don't use bare yield, only use yield from, and the latter always with either a Future or a coroutine as argument); - either of the above, which is actually the most common requirement -- most asyncio functions that support one also support the other, and either is allowable as the argument to `yield from`. In the implementation we so often flipped between Future and coroutine that I imagine sometimes the implementation and docs differ; also, we don't have a good short name for either of the above so we end up using one or the other as a shorthand. OK, that makes a lot of sense. *Unless* you want to attach callbacks, inspect the result or exception, or cancel it (all of which require a Future), your code shouldn't be concerned about the difference -- you should just use `res = yield from func(args)` and use try/except to catch exceptions if you care. And if you do need a Future, you can call the function asyncio.async() on it (which in PEP 492 is renamed to ensure_future()). Again, makes sense. Although there are some bits of example code in the docs that call asyncio.async() on a coroutine and throw away the result (for example, https://docs.python.org/3/library/asyncio-task.html#example-future-with-run-until-complete ). That confuses me. Are you saying that async() modifies its (coroutine) argument to make it a Future? Rather than wrapping a coroutine in a Future, which gets returned? No, it wraps a coroutine (i.e. a generator) in a Task, but leaves a Future alone. I'm stumped why that example calls async() and then throws the result away. I suspect it won't work without it (or else Victor wouldn't have added the call) but the reason seems, um, deep. I think wrapping it in a Task enters the generator in the event loop's queue of runnables -- otherwise the generator may well be garbage-collected without ever running. Such complexity doesn't belong in such a simple example though. In the PEP 492 world, these concepts map as follows: - Future translates to something with an __await__ method (and asyncio Futures are trivially made compliant by defining Future.__await__ as an alias for Future.__iter__); - asyncio coroutine maps to PEP 492 coroutine object (either defined with `async def` or a generator decorated with @types.coroutine -- note that @asyncio.coroutine incorporates the latter); - either of the above maps to awaitable. OK. Although future is a nicer term than something with an __await__ method and the plethora of flavours of coroutine is not great. But given that the only term we'll need in common cases is awaitable, it's still a net improvement. So in the PEP 492 world, there's no such thing as a Task outside of asyncio? Or, to put it another way, a Task is only relevant in an IO context (unless an alternative event loop library implemented a similar concept), and we should only be talking in terms of awaitables and futures (given concurrent.futures and asyncio, I doubt you're going to be able to stop people using Future for the generic term for something with an __await__ method at best, and quite possibly as equivalent to awaitable, unfortunately). I'm not sure. But it's true that Futures and Tasks in asyncio serve the purpose of linking the event loop (whose basic functioning is callback-based) to coroutines (implemented by generators). The basic idea is that when some I/O completes the event loop will call a callback function registered for that particular I/O operation; the callback then is actually a bound method of a Future or Task that causes the latter to be marked as complete (i.e. having a result) which in turn will call other callbacks (registered with the Future using add_done_callback()); in the case of a Task (i.e. a special kind of Future that wraps a generator/coroutine) this will resume the coroutine. (Actually it may resume an entire stack of coroutines that are blocked waiting for each other at yield-from; in my spare time I'm working on an explanation of the machinery underlying yield, yield
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 5 May 2015 at 21:00, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. Given that the stdlib doesn't provide an example of such a UI loop, what would a 3rd party module need to implement to provide such a thing? Can any of the non-IO related parts of asyncio be reused for the purpose, or must the 3rd party module implement everything from scratch? To me, this is an important question, as it cuts directly to the heart of the impression people have that coroutines and async are only for asyncio. I'd be interested in writing, for instructional purposes, a toy but complete event loop. But I'm *not* really interested in trying to reverse engineer the required interface. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: Please mention the Event Loop
On Tue May 5 21:44:26 CEST 2015,Brett Cannon wrote: It's not as complicated as it seems when you realize there is an event loop driving everything (which people have been leaving out of the conversation since it doesn't tie into the syntax directly). Another reason people don't realize it is that the PEP goes out of its way to avoid saying so. I understand that you (and Yuri) don't want to tie the PEP too tightly to the specific event loop implementation in asyncio.events.AbstractEventLoop, but ... that particular conflation isn't really what people are confused about. coroutines often brings up thoughts of independent tasks. Yuri may well know that (Python has asymmetric coroutines, that's it), but others have posted that this was a surprise -- and the people posting here have far more python experience than most readers will. Anyone deeply involved enough to recognize that this PEP is only about (1) a particular type of co-routine -- a subset even of prior python usage (2) used for a particular purpose (3) coordinated via an external scheduler will already know that they can substitute other event loops. Proposed second paragraph of the abstract: This PEP assumes that the asynchronous tasks are scheduled and coordinated by an Event Loop similar to that of stdlib module asyncio.events.AbstractEventLoop. While the PEP is not tied to any specific Event Loop implementation, it is relevant only to the kind of coroutine that uses yield as a signal to the scheduler, indicating that the coroutine will be waiting until an event (such as IO) is completed. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
On 5 May 2015 at 21:57, Guido van Rossum gu...@python.org wrote: On Tue, May 5, 2015 at 1:39 PM, Paul Moore p.f.mo...@gmail.com wrote: It's very hard to separate coroutines from asyncio, because there's no other example (not even a toy one) to reason about. What about Greg Ewing's example? http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Scheduler/scheduler.txt That doesn't cover any of the higher level abstractions like tasks or futures (at least not by those names or with those interfaces). And I don't see where the PEP 492 additions would fit in (OK, replace yield from with await is part of it, but I don't see the rest). We may be talking at cross purposes here. There's a lot of asyncio that doesn't seem to me to be IO-related. Specifically the future and task abstractions. I view those as relevant to coroutine programming in Python because they are referenced in any discussion of coroutines (you yield from a future, for example). If you see them as purely asyncio related (and not directly usable from outside of an asyncio context) then that may explain some of my confusion (but at the cost of reducing the coroutine concept to something pretty trivial in the absence of a library that independently implements these concepts). In some ways I wish there had been an asyncio library that covered the areas that are fundamentally about IO multiplexing. And a separate library (just async, maybe, although that's now a bad idea as it clashes with a keyword :-)) that covered generic event loop, task and synchronisation areas. But that's water under the bridge now. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sub-claasing pathlib.Path seems impossible
In article caab4jgkqylh4epgs2tnty1rmaddig4wrojzx9ek8zepxtwq...@mail.gmail.com, Christophe Bal projet...@gmail.com wrote: In this post http://stackoverflow.com/questions/29850801/simple-subclassing-pathlib-path-d oes-not-work/29854141#29854141, I have noticed a problem with the following code. [...] This breaks the sub-classing from Python point of view. In the post http://stackoverflow.com/questions/29850801/simple-subclassing-pathlib-path-d oes-not-work/29854141#29854141, I give a hack to sub-class Path but it's a bit Unpythonic. In article cap7+vjlkrtavb8bp34wwtdkx0ra+opb1tz+9ojkkwlqwtup...@mail.gmail.com, Guido van Rossum gu...@python.org wrote: It does sound like subclassing Path should be made easier. Christophe, if you want to pursue this, you should open an issue for it on the Python bug tracker, bugs.python.org. Otherwise, it will likely be forgotten here. -- Ned Deily, n...@acm.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On Tue, May 5, 2015 at 2:01 PM, Paul Moore p.f.mo...@gmail.com wrote: On 5 May 2015 at 21:00, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. Given that the stdlib doesn't provide an example of such a UI loop, what would a 3rd party module need to implement to provide such a thing? Can any of the non-IO related parts of asyncio be reused for the purpose, or must the 3rd party module implement everything from scratch? To me, this is an important question, as it cuts directly to the heart of the impression people have that coroutines and async are only for asyncio. I'd be interested in writing, for instructional purposes, a toy but complete event loop. But I'm *not* really interested in trying to reverse engineer the required interface. This is a great idea. What kind of application do you have in mind? I think the main real-life use case for using coroutines with a UI event loop is newer Windows code. C# (and IIUC VB) has coroutines very much along the lines of PEP 492, and all code that does any kind of I/O (whether disk or network) must be written as a coroutine. This requirement is enforced by the C# compiler: the basic system calls for doing I/O are coroutines, and in order to get their result you must use an await expression, which in turn may only be used in a coroutine. Thus all code that may invoke an I/O call ends up being a coroutine. This is exactly the type of constraint we're trying to introduce into Python with PEP 492 (except of course we're not making all I/O primitives coroutines -- that would be madness, we're going with optional instead). -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 4
Tue May 5 21:48:36 CEST 2015, Yury Selivanov wrote: As for terminology, I view this discussion differently. It's not about the technical details (Python has asymmetric coroutines, that's it), but rather on how to disambiguate coroutines implemented with generators and yield-from, from new 'async def' coroutines. Not just How?, but Why?. Why do they *need* to be disambiguated? With the benefit of having recently read all that discussion (as opposed to just the PEP), my answer is ... uh ... that generators vs async def is NOT an important distinction. What matters (as best I can tell) is: something using yield (or yield from) to mark execution context switches vs other kinds of callables, including those using yield to make an iterator I'm not quite sure that the actual proposal even really separates them effectively, in part because the terminology keeps suggesting other distinctions instead. (The glossary does help; just not enough.) -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 2015-05-05 5:01 PM, Paul Moore wrote: On 5 May 2015 at 21:00, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. Given that the stdlib doesn't provide an example of such a UI loop, what would a 3rd party module need to implement to provide such a thing? Can any of the non-IO related parts of asyncio be reused for the purpose, or must the 3rd party module implement everything from scratch? The idea is that you integrate processing of UI events to your event loop of choice. For instance, Twisted has integration for QT and other libraries [1]. This way you can easily combine async network (or OS) calls with your UI logic to avoid callback hell. Quick search for something like that for asyncio revealed this library: [2]. This small library actually re-implements relevant low-level parts of the asyncio event loop on top of QT primitives (another approach). Yury [1] http://twistedmatrix.com/trac/wiki/QTReactor [2] https://github.com/harvimt/quamash#usage -- see first_50 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 5 May 2015 at 22:25, Guido van Rossum gu...@python.org wrote: I'd be interested in writing, for instructional purposes, a toy but complete event loop. But I'm *not* really interested in trying to reverse engineer the required interface. This is a great idea. What kind of application do you have in mind? At this point, *all* I'm thinking of is a toy. So, an implementation somewhat parallel to asyncio, but where the event loop just passes control to the next task - so no IO multiplexing. Essentially Greg Ewing's example up to, but not including, Waiting for External Events. And ideally I'd like to think that Waiting for Resources can be omitted in favour of reusing https://docs.python.org/3/library/asyncio-sync.html and https://docs.python.org/3/library/asyncio-queue.html. My fear is, however, that those parts of asyncio aren't reusable for other event loops, and every event loop implementation has to reinvent those wheels. When I say the required interface I'm thinking in terms of what's needed to allow reuse of the generic parts of asyncio. If nothing of asyncio is generic in those terms, then the exercise will be futile (except in the negative sense of confirming that there are no reusable async components in the stdlib). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: async/await in Python; version 5
On 5 May 2015 at 22:38, Yury Selivanov yselivanov...@gmail.com wrote: n 2015-05-05 5:01 PM, Paul Moore wrote: On 5 May 2015 at 21:00, Yury Selivanov yselivanov...@gmail.com wrote: On 2015-05-05 3:40 PM, Jim J. Jewett wrote: On Tue May 5 18:29:44 CEST 2015, Yury Selivanov posted an updated PEP492. Where are the following over-simplifications wrong? (1) The PEP is intended for use (almost exclusively) with asychronous IO and a scheduler such as the asynchio event loop. Yes. You can also use it for UI loops. Basically, anything that can call your code asynchronously. Given that the stdlib doesn't provide an example of such a UI loop, what would a 3rd party module need to implement to provide such a thing? Can any of the non-IO related parts of asyncio be reused for the purpose, or must the 3rd party module implement everything from scratch? The idea is that you integrate processing of UI events to your event loop of choice. For instance, Twisted has integration for QT and other libraries [1]. This way you can easily combine async network (or OS) calls with your UI logic to avoid callback hell. We seem to be talking at cross purposes. You say the PEP is *not* exclusively intended for use with asyncio. You mention UI loops, but when asked how to implement such a loop, you say that I integrate UI events into my event loop of choice. But what options do I have for my event loop of choice? Please provide a concrete example that isn't asyncio. Can I use PEP 492 with Twisted (I doubt it, as Twisted doesn't use yield from, which is Python 3.x only)? I contend that there *is* no concrete example that currently exists, so I'm asking what I'd need to do to write one. You pointed at qamash, but that seems to be subclassing asyncio, so isn't something that isn't asyncio. Note that I don't have a problem with there being no existing implementation other than asyncio. I'd just like it if we could be clear over exactly what we mean when we say the PEP is not tied to asyncio. It feels like the truth currently is you can write your own async framework that uses the new features introduced by the PEP. I fully expect that *if* there's a need for async frameworks that aren't fundamentally IO multiplexors, then it'll get easier to write them over time (the main problem right now is a lack of good tutorial examples of how to do so). But at the moment, asyncio seems to be the only game in town (and I can imagine that it'll always be the main IO multiplexor, unless existing frameworks like Twisted choose to compete rather than integrate). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: async/await in Python; version 4
Hi, still watching progress here. Read all posts and changes. Everything improved and I know it is a lot of work. Thx for doing this. But I still think this PEP goes to far. 1. To have native coroutines and await, __await__ is very good and useful. Also for beginners to see and mark coroutines are a different concept than generators. Also this is easy to learn. Easier as to explain why a generator is in this case a coroutine and so on. 2. I still don't like to sprinkle async everywhere. At all we don't need it for the first step. We can handle coroutines similar to generators, when there is a await in it then it is one. Same as for yield. Or to be more explicit, if it is marked as one with @coroutine it is enough. But then it makes also sense to do the same for generators with @generator. We should be consistent here. 3. async with is not needed, there are rare use cases for it and every can be written with a try: except: finally: Every async framework lived without it over years. No problem because for the seldom need try: ... was enough. 4. async for can be implemented with a while loop. For me this is even more explicit and clear. Every time I see the async for I try to find out what is done in async manner. For what I do an implicit await ? Also in most of my uses cases it was enough to produce Futures in a normal loop and yield (await) for them. Even for database interfaces. Most of the time they do prefetching under the hood and I don't have to care at all. 5. For async with/for a lot of new special methods are needed all only prefixed with a. Easy to confuse with a for abstract. Often used to prefix abstract classes. Still think __async_iter__, __async_enter__ is better for this. Yes more to write but not so easy to confuse with the sync __iter__ or __enter__, ... And matches more to I must use async to call them. I am not new to the async land, have done a lot of stuff with twisted and even looked at tornado. Also has tried to teach some people the asnc world. This is not easy and you learn most only by doing and using it. For my final conclusion, we should not rush all this into the language. Do it step by step and also help other Python implementations to support it. For now only a very low percentage are at Python 3.4. And if you compare the statistics in PyPi, you see most are still at Python 2.7 and using async frameworks like twisted/tornado. We do such a fundamental language change only because a few people need it for a niche in the whole Python universe? We confuse the rest with new stuff they never need? Even the discussion on python-dev suggests there is some time needed to finalize all this. We forget to address the major problems here. How can someone in a sync script use this async stuff easy. How can async and sync stuff cooperate and we don't need to rewrite the world for async stuff. How can a normal user access the power of async stuff without rewriting all his code. So he can use a simple asyc request library in his code. How can a normal user learn and use all this in an easy way. And for all this we still can't tell them oh the async stuff solves the multiprocessing problem of Python learn it and switch to version 3.5. It does not and it is only most useful for networking stuff nothing more. Don't get me wrong, I like async programming and I think it is very useful. But had to learn not everyone thinks so and most only want to solve there problems in an easy way and not get a new one called async. Now I shut up. Go to my normal mode and be quiet and read. ;-) Regards, Wolfgang ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
