Re: [Python-Dev] PEP 492: No new syntax is required
Hello, On Mon, 27 Apr 2015 08:48:49 +0100 Mark Shannon m...@hotpy.org wrote: On 27/04/15 00:13, Guido van Rossum wrote: But new syntax is the whole point of the PEP. I want to be able to *syntactically* tell where the suspension points are in coroutines. Doesn't yield from already do that? Currently this means looking for yield [from]; PEP 492 just adds looking for await and async [for|with]. Making await() a function defeats the purpose because now aliasing can hide its presence, and we're back in the land of gevent or stackless (where *anything* can potentially suspend the current task). I don't want to live in that land. I don't think I was clear enough. I said that await *is* a function, not that is should be disguised as one. Yes, you said, but it is not. I guess other folks left figuring that out for yourself, and it's worthy exercise. Hint: await appears to translate to GET_AWAITABLE and YIELD_FROM opcodes. If your next reply is I told you so, then you again miss that await is a special Python language construct (effectively, operator), while the fact that its implemented as GET_AWAITABLE and YIELD_FROM opcodes in CPython is only CPython's implementation detail, CPython being just one (random) Python language implementation. Reading the code, GetAwaitableIter would be a better name for that element of the implementation. It is a straightforward non-blocking function. Based on all this passage, my guess is that you miss difference between C and Python functions. On C level, there're only functions used to implement everything (C doesn't offer anything else). But on Python level, there're larger variety: functions, methods, special forms (a term with a bow to Scheme - it's a function which you can't implement in terms of other functions and which may have behavior they can't have). await is a special form. The fact that it's implemented by a C function (or not exactly, as pointed above) is just CPython's implementation detail. Arguing that await should be something based on what you saw in C code is putting it all backwards. -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Hello, On Tue, 28 Apr 2015 20:59:18 +0100 Mark Shannon m...@hotpy.org wrote: On 28/04/15 20:24, Paul Sokolovsky wrote: Hello, [snip] Based on all this passage, my guess is that you miss difference between C and Python functions. This is rather patronising, almost to the point of being insulting. Please keep the debate civil. And yet people do make mistakes and misunderstand, and someone should bother with social psychology of programming - why this happens, what are typical patterns, root causes, etc. I don't think you should be insulted, especially if you think you're right with your points - then all counter arguments will either help you understand another side, or will just look funny. [snip] Cheers, Mark. -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On 28/04/15 20:24, Paul Sokolovsky wrote: Hello, [snip] Based on all this passage, my guess is that you miss difference between C and Python functions. This is rather patronising, almost to the point of being insulting. Please keep the debate civil. [snip] Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On Tue, Apr 28, 2015 at 11:51 AM, Stefan Behnel stefan...@behnel.de wrote: Mark Shannon schrieb am 27.04.2015 um 09:48: On 27/04/15 00:13, Guido van Rossum wrote: Currently this means looking for yield [from]; PEP 492 just adds looking for await and async [for|with]. Making await() a function defeats the purpose because now aliasing can hide its presence, and we're back in the land of gevent or stackless (where *anything* can potentially suspend the current task). I don't want to live in that land. I don't think I was clear enough. I said that await *is* a function, not that is should be disguised as one. Reading the code, GetAwaitableIter would be a better name for that element of the implementation. It is a straightforward non-blocking function. 1) it's not like people commonly alias repr() or len(), so why would they alias an await() builtin ? Unless, obviously, there's an actual reason to do so, in which case having it as a functions comes in handy. :) We had the same line of reasoning with print() back in the days of Py3k. 2) an await() builtin function that calls an __await__() special method on its input object sounds very pythonic. This sounds confused. The await expression must be recognized by the parser so it can generate different code for it (the code to suspend the stack). A builtin function cannot generate different code -- to the compiler all functions look the same. I know we could change that rule, but that' would be a really a big deviation from Python's philosophy: Currently the code generator never needs to know the type of any variables -- and a builtin function 'await' would just be another variable to the code generator. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On 4/26/2015 4:32 PM, Paul Sokolovsky wrote: Then, is the only logic for proposing __aenter__ is to reinsure against a situation that someone starts to write async context manager, forgets that they write async context manager, and make an __enter__ method there. Then your implementation will announce that async context manager lacks __aenter__, whereas my approach would announce Async's manager __enter__ did not return awaitable value. Again, is that the distinction you're shooting for, or do I miss something? Seems like the missing __aenter__ can easily be detected by the interpreter at compile time, but the wrong type returned would be at run time, or after a complex type-analysis done at compile time (unlikely to be practical). So I think you've nailed the distinction... but I'm not the expert. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Hello, On Sun, 26 Apr 2015 16:40:03 -0400 Yury Selivanov yselivanov...@gmail.com wrote: Hi Mark, On 2015-04-26 4:21 PM, Mark Shannon wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Mark, all your points are explained in the PEP in a great detail: Indeed, they are. It is the very basic counter-argument against this PEP is that everything it proposes is already doable. But the PEP makes very clear that the only reason it exists is to make coroutine programming easier and less error-prone. However, given that there're questions even like that and you're great at keeping up discussion, I'd appreciate additional argumentation on: 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? Because you can't reuse them. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-magic-names Ok, so here're 3 points this link gives, with my concerns/questions: An alternative idea about new asynchronous iterators and context managers was to reuse existing magic methods, by adding an async keyword to their declarations: [But:] - it would not be possible to create an object that works in both with and async with statements; Yes, and I would say, for good. Behavior of sync and async code is different enough to warrant separate classes (if not libraries!) to implement each paradigm. What if, in otherwise async code, someone will miss async in async with and call sync version of context manager? So, losing ability stated above isn't big practical loss. - it would look confusing Sorry, async def __enter__ doesn't look more confusing than __aenter__ (vs __enter__). and would require some implicit magic behind the scenes in the interpreter; Interpreter already does a lot of implicit magic. Can you please elaborate what exactly would need to be done? one of the main points of this proposal is to make coroutines as simple and foolproof as possible. And it's possible to agree that to separate notions, create a dichotomy is a simple principle on its own. But just taking bunch of stuff - special methods, exceptions - and duplicating it is a bloat a violates another principle - DRY. You argue that this will make coroutine writing simple. But coroutines are part of the language, and duplicating notions makes language more complex/complicated. Can you please argue that in this case it's worth duplicating hierarchies instead of reusing existing lower-level concepts, given that higher-level concepts are already distinguished well enough (async and await keywords separate old and new things very visibly). -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Hello, On Sun, 26 Apr 2015 18:49:43 -0400 Yury Selivanov yselivanov...@gmail.com wrote: [] - it would look confusing Sorry, async def __enter__ doesn't look more confusing than __aenter__ (vs __enter__). I'll update the PEP. The argument shouldn't be that it's confusing, the argument is that __aenter__ returns an 'awaitable', which is either a coroutine-object or a future. You can't reuse __enter__, because you'd break backwards compatibility -- it's perfectly normal for context managers in python to return any object from their __enter__. If we assign some special meaning to futures -- we'll break existing code. So, again to make sure I (and hopefully other folks) understand it right. You say it's perfectly normal for context managers in python to return any object from their __enter__. That's true, but we talk about async context managers. There're no such at all, they yet need to be written. And whoever writes them, would need to return from __enter__ awaitable, because that's the requirement for an async context manager, and it is error to return something else. Then, is the only logic for proposing __aenter__ is to reinsure against a situation that someone starts to write async context manager, forgets that they write async context manager, and make an __enter__ method there. Then your implementation will announce that async context manager lacks __aenter__, whereas my approach would announce Async's manager __enter__ did not return awaitable value. Again, is that the distinction you're shooting for, or do I miss something? [] Anyways, I really doubt that you can convince anyone to reuse existing dunder methods for async stuff. Yeah, but it would be nice to understand why everyone and so easily agrees to them, after pretty thorough discussion of other aspects. Thanks, Yury -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Hello, On Sun, 26 Apr 2015 16:13:43 -0700 Guido van Rossum gu...@python.org wrote: But new syntax is the whole point of the PEP. I want to be able to *syntactically* tell where the suspension points are in coroutines. Currently this means looking for yield [from]; PEP 492 just adds looking for await and async [for|with]. Making await() a function defeats the purpose because now aliasing can hide its presence, and we're back in the land of gevent or stackless (where *anything* can potentially suspend the current task). I don't want to live in that land. And someone in this thread should have link in a signature to the page on why there's not wanting to live in that land. I don't have it handy, so may offer only reminiscence: if you don't know where there're suspension points, you essentially should assume that any function call can be a suspension point. And then you dropped almost as low as when using threads - you can lose control anytime, data may change anytime, you need to extensively use locks, etc. Oh, and btw, there was related claim earlier that some things are better done in thread pools, so async context managers can be done away with. That's another faulty point - if asyncio is to be feature-complete, it should work [well] with own functionality, without crutches of foreign paradigms. Otherwise you may be just happy to use threads for everything at all, like everyone did 5 years ago. (And yeah, there's even usages for that, for example, MicroPython is proudly GIL-free, translating to not supporting those thread thingies and relying on native Python concurrency primitives). On Sun, Apr 26, 2015 at 1:21 PM, Mark Shannon m...@hotpy.org wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. Please don't add unnecessary new syntax. Cheers, Mark. P.S. I'm not objecting to any of the other new features proposed, just the new syntax. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Hello, On Sun, 26 Apr 2015 19:45:30 -0400 Yury Selivanov yselivanov...@gmail.com wrote: [] Then, is the only logic for proposing __aenter__ is to reinsure against a situation that someone starts to write async context manager, forgets that they write async context manager, and make an __enter__ method there. It's to make sure that it's impossible to accidentally use existing regular context manager that returns a future object from its __enter__ / __exit__ (nobody prohibits you to return a future object from __exit__, although it's pointless) in an 'async with' block. I see, so it's just to close the final loophole, unlikely to be hit in real life (unless you can say that there're cases of doing just that in existing asyncio libs). Well, given that Greg Ewing wanted even stricter error-proofness, and you rejected it as such strict as to disallow useful behavior, I've just got to trust you that in this case, you're as strict as needed. I really don't understand the desire to reuse existing magic methods. Even if it was decided to reuse them, it wouldn't even simplify the implementation in CPython; the code there is already DRY (I don't re-implement opcodes for 'with' statement; I reuse them). Well, there're 3 levels of this stuff: 1. How mere people write their code - everyone would use async def and await, this should be bullet- (and fool-) proof. 2. How library code is written - async iterators won't be written by everyone, and only few will write async context managers; it's fair to expect that people doing these know what they do and don't do stupid mistakes. 3. How it all is coded in particular Python implementation. It's clear that __enter__ vs __aenter__ distinction is 1st kind of issue in your list. As for 3rd point, I'd like to remind that CPython is only one Python implementation. And with my MicroPython hat on, I'd like to know if (some of) these new features are bloat or worthy for the space constraints we have. I appreciate the answers you gave on all accounts! Thanks! Yury -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Hello, On Sun, 26 Apr 2015 20:39:55 -0400 Yury Selivanov yselivanov...@gmail.com wrote: [] As for 3rd point, I'd like to remind that CPython is only one Python implementation. And with my MicroPython hat on, I'd like to know if (some of) these new features are bloat or worthy for the space constraints we have. OT: MicroPython is an amazing project. Kudos for doing it. I really hope that addition of few new magic methods won't make it too hard for you guys to implement PEP 492 in MicroPython one day. Thanks! Damien George, MicroPython's author, actually already made a basic implementation of async def/await: https://github.com/micropython/micropython/commit/81afa7e098634605c04597d34a51ca2e59a87d7c So we surely do hope this PEP will be accepted, and soonish! ;-) -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Mark Shannon schrieb am 27.04.2015 um 09:48: On 27/04/15 00:13, Guido van Rossum wrote: Currently this means looking for yield [from]; PEP 492 just adds looking for await and async [for|with]. Making await() a function defeats the purpose because now aliasing can hide its presence, and we're back in the land of gevent or stackless (where *anything* can potentially suspend the current task). I don't want to live in that land. I don't think I was clear enough. I said that await *is* a function, not that is should be disguised as one. Reading the code, GetAwaitableIter would be a better name for that element of the implementation. It is a straightforward non-blocking function. 1) it's not like people commonly alias repr() or len(), so why would they alias an await() builtin ? Unless, obviously, there's an actual reason to do so, in which case having it as a functions comes in handy. :) We had the same line of reasoning with print() back in the days of Py3k. 2) an await() builtin function that calls an __await__() special method on its input object sounds very pythonic. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On 27/04/15 00:13, Guido van Rossum wrote: But new syntax is the whole point of the PEP. I want to be able to *syntactically* tell where the suspension points are in coroutines. Doesn't yield from already do that? Currently this means looking for yield [from]; PEP 492 just adds looking for await and async [for|with]. Making await() a function defeats the purpose because now aliasing can hide its presence, and we're back in the land of gevent or stackless (where *anything* can potentially suspend the current task). I don't want to live in that land. I don't think I was clear enough. I said that await *is* a function, not that is should be disguised as one. Reading the code, GetAwaitableIter would be a better name for that element of the implementation. It is a straightforward non-blocking function. On Sun, Apr 26, 2015 at 1:21 PM, Mark Shannon m...@hotpy.org mailto:m...@hotpy.org wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. Please don't add unnecessary new syntax. Cheers, Mark. P.S. I'm not objecting to any of the other new features proposed, just the new syntax. ___ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido http://python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On 26/04/15 23:24, Nick Coghlan wrote: On 27 Apr 2015 07:50, Mark Shannon m...@hotpy.org mailto:m...@hotpy.org wrote: On 26/04/15 21:40, Yury Selivanov wrote: But it's hard. Iterating through something asynchronously? Write a 'while True' loop. Instead of 1 line you now have 5 or 6. Want to commit your database transaction? Instead of 'async with' you will write 'try..except..finally' block, with a very high probability to introduce a bug, because you don't rollback or commit properly or propagate exception. I don't see why you can't do transactions using a 'with' statement. Because you need to pass control back to the event loop from the *__exit__* method in order to wait for the commit/rollback operation without blocking the scheduler. The with (yield from cm()) formulation doesn't allow either __enter__ *or* __exit__ to suspend the coroutine to wait for IO, so you have to do the IO up front and return a fully synchronous (but still non-blocking) CM as the result. True. The 'with' statement cannot support this use case, but try-except can do the job: trans = yield from db_conn.transaction() try: ... except: yield from trans.roll_back() raise yield from trans.commit() Admittedly not as elegant as the 'with' statement, but perfectly readable. We knew about these problems going into PEP 3156 (http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programming.html#using-special-methods-in-explicitly-asynchronous-code) so it's mainly a matter of having enough experience with asyncio now to be able to suggest specific syntactic sugar to make the right way and the easy way the same way. asyncio is just one module amongst thousands, does it really justify special syntax? Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On Sun, Apr 26, 2015, 17:49 Mark Shannon m...@hotpy.org wrote: On 26/04/15 21:40, Yury Selivanov wrote: Hi Mark, On 2015-04-26 4:21 PM, Mark Shannon wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Mark, all your points are explained in the PEP in a great detail: I did read the PEP. I do think that clarifying the distinction between coroutines and 'normal' generators is a good idea. Adding stuff to the standard library to help is fine. I just don't think that any new syntax is necessary. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. Yes, everything that the PEP proposes can be done without new syntax. That's how people use asyncio right now, with only what we have in 3.4. But it's hard. Iterating through something asynchronously? Write a 'while True' loop. Instead of 1 line you now have 5 or 6. Want to commit your database transaction? Instead of 'async with' you will write 'try..except..finally' block, with a very high probability to introduce a bug, because you don't rollback or commit properly or propagate exception. I don't see why you can't do transactions using a 'with' statement. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. https://www.python.org/dev/peps/pep-0492/#rationale-and-goals states that it is not possible to natively define a coroutine which has no yield or yield from statement which is just not true. https://www.python.org/dev/peps/pep-0492/#debugging-features Requires the addition of the CO_COROUTINE flag, not any new keywords. https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword Seems to be repeating the above. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? Because you can't reuse them. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-for-and-with-statements Which seems back to front. The argument is that existing syntax constructs cannot be made to work with asynchronous objects. Why not make the asynchronous objects work with the existing syntax? https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-magic-names The argument here relies on the validity of the previous points. 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. It can't be replaced by a function. Only if you use greenlets or Stackless Python. Why not? The implementation of await is here: https://github.com/python/cpython/compare/master...1st1:await#diff-23c87bfada1d01335a3019b9321502a0R642 which clearly could be made into a function. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. There is no equivalent to 'async with'. with (yield from xxx) only allows you to suspend execution in __enter__ (and it's not actually in __enter__, but in a coroutine that returns a context manager). https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with see New Syntax section to see what 'async with' is equivalent too. Which, by comparing with PEP 343, can be translated as: with expr as e: e = await(e) ... Please don't add unnecessary new syntax. It is necessary. This isn't an argument, it's just contradiction ;) Perhaps you haven't spent a lot of time maintaining huge code-bases developed with frameworks like asyncio, so I understand why it does look unnecessary to you. This is a good reason for clarifying the distinction between 'normal' generators and coroutines. It is not, IMO, justification for burdening the language (and everyone porting Python 2 code) with extra syntax. How is it a burden for people porting Python 2 code? Because they won't get to name anything 'async' just like anyone supporting older Python 3 versions? Otherwise I don't see how it is of any consequence to people maintaining 2/3 code as it will just be another thing they can't use until they drop Python 2 support. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On 2015-04-26 5:48 PM, Mark Shannon wrote: On 26/04/15 21:40, Yury Selivanov wrote: Hi Mark, On 2015-04-26 4:21 PM, Mark Shannon wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Mark, all your points are explained in the PEP in a great detail: I did read the PEP. I do think that clarifying the distinction between coroutines and 'normal' generators is a good idea. Adding stuff to the standard library to help is fine. I just don't think that any new syntax is necessary. Well, unfortunately, I can't explain why the new syntax is necessary better than it is already explained in the PEP, sorry. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. Yes, everything that the PEP proposes can be done without new syntax. That's how people use asyncio right now, with only what we have in 3.4. But it's hard. Iterating through something asynchronously? Write a 'while True' loop. Instead of 1 line you now have 5 or 6. Want to commit your database transaction? Instead of 'async with' you will write 'try..except..finally' block, with a very high probability to introduce a bug, because you don't rollback or commit properly or propagate exception. I don't see why you can't do transactions using a 'with' statement. Because you can't use 'yield from' in __exit__. And allowing it there isn't a very good idea. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. https://www.python.org/dev/peps/pep-0492/#rationale-and-goals states that it is not possible to natively define a coroutine which has no yield or yield from statement which is just not true. It *is* true. Please note the word natively. See coroutine decorator from asyncio: https://github.com/python/cpython/blob/master/Lib/asyncio/coroutines.py#L130 To turn a regular function into a coroutine you have three options: 1. wrap the function; 2. use if 0: yield terrible hack; 3. flip CO_GENERATOR flag for the code object (which is CPython implementation detail); also terrible hack. @coroutine decorator is great because we can use asyncio even in 3.3. But it's a) easy to forget b) hard to statically analyze c) hard to explain why functions decorated with it must only contain 'yield from' instead of 'yield' d) few more reasons listed in the PEP https://www.python.org/dev/peps/pep-0492/#debugging-features Requires the addition of the CO_COROUTINE flag, not any new keywords. True. But the importance of new keywords is covered in other sections. https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword Seems to be repeating the above. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? Because you can't reuse them. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-for-and-with-statements Which seems back to front. The argument is that existing syntax constructs cannot be made to work with asynchronous objects. Why not make the asynchronous objects work with the existing syntax? Because a) it's not possible; b) the point is to make suspend points visible in the code. That's one of the key principles of asyncio and other frameworks. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-magic-names The argument here relies on the validity of the previous points. 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. It can't be replaced by a function. Only if you use greenlets or Stackless Python. Why not? The implementation of await is here: https://github.com/python/cpython/compare/master...1st1:await#diff-23c87bfada1d01335a3019b9321502a0R642 which clearly could be made into a function. Implementation of 'await' requires YIELD_FROM opcode. As far as I know functions in Python can't push opcodes to the eval loop while running. The only way to do what you propose is to use greenlets or merge stackless into cpython. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. There is no equivalent to 'async with'. with (yield from xxx) only allows you to suspend execution in __enter__ (and it's not actually in __enter__, but in a coroutine that returns a context manager). https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with see New Syntax section to see what 'async with' is equivalent too. Which, by comparing with PEP 343, can be translated as: with expr as e: e = await(e) ... Please don't add unnecessary new syntax. It is necessary. This isn't an argument, it's just contradiction ;) Perhaps you haven't spent a lot of time maintaining huge code-bases developed with frameworks like asyncio,
Re: [Python-Dev] PEP 492: No new syntax is required
Paul, On 2015-04-26 7:32 PM, Paul Sokolovsky wrote: Hello, On Sun, 26 Apr 2015 18:49:43 -0400 Yury Selivanov yselivanov...@gmail.com wrote: [] - it would look confusing Sorry, async def __enter__ doesn't look more confusing than __aenter__ (vs __enter__). I'll update the PEP. The argument shouldn't be that it's confusing, the argument is that __aenter__ returns an 'awaitable', which is either a coroutine-object or a future. You can't reuse __enter__, because you'd break backwards compatibility -- it's perfectly normal for context managers in python to return any object from their __enter__. If we assign some special meaning to futures -- we'll break existing code. So, again to make sure I (and hopefully other folks) understand it right. You say it's perfectly normal for context managers in python to return any object from their __enter__. That's true, but we talk about async context managers. There're no such at all, they yet need to be written. And whoever writes them, would need to return from __enter__ awaitable, because that's the requirement for an async context manager, and it is error to return something else. Then, is the only logic for proposing __aenter__ is to reinsure against a situation that someone starts to write async context manager, forgets that they write async context manager, and make an __enter__ method there. It's to make sure that it's impossible to accidentally use existing regular context manager that returns a future object from its __enter__ / __exit__ (nobody prohibits you to return a future object from __exit__, although it's pointless) in an 'async with' block. I really don't understand the desire to reuse existing magic methods. Even if it was decided to reuse them, it wouldn't even simplify the implementation in CPython; the code there is already DRY (I don't re-implement opcodes for 'with' statement; I reuse them). Then your implementation will announce that async context manager lacks __aenter__, whereas my approach would announce Async's manager __enter__ did not return awaitable value. Again, is that the distinction you're shooting for, or do I miss something? [] Anyways, I really doubt that you can convince anyone to reuse existing dunder methods for async stuff. Yeah, but it would be nice to understand why everyone and so easily agrees to them, after pretty thorough discussion of other aspects. NP :) FWIW, I wasn't trying to dodge the question, but rather stressing that the DRY argument is weak. And thanks for pointing out that this isn't covered in the PEP in enough detail. I'll update the PEP soon. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Brett, On 2015-04-26 6:09 PM, Brett Cannon wrote: How is it a burden for people porting Python 2 code? Because they won't get to name anything 'async' just like anyone supporting older Python 3 versions? Otherwise I don't see how it is of any consequence to people maintaining 2/3 code as it will just be another thing they can't use until they drop Python 2 support. Agree. Also, the PEP states that we'll either keep a modified tokenizer or use __future__ imports till at least 3.7. It's *3 or more years*. And if necessary, we can wait till 3.8 (python 2 won't be supported at that point). Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On 27 Apr 2015 07:50, Mark Shannon m...@hotpy.org wrote: On 26/04/15 21:40, Yury Selivanov wrote: But it's hard. Iterating through something asynchronously? Write a 'while True' loop. Instead of 1 line you now have 5 or 6. Want to commit your database transaction? Instead of 'async with' you will write 'try..except..finally' block, with a very high probability to introduce a bug, because you don't rollback or commit properly or propagate exception. I don't see why you can't do transactions using a 'with' statement. Because you need to pass control back to the event loop from the *__exit__* method in order to wait for the commit/rollback operation without blocking the scheduler. The with (yield from cm()) formulation doesn't allow either __enter__ *or* __exit__ to suspend the coroutine to wait for IO, so you have to do the IO up front and return a fully synchronous (but still non-blocking) CM as the result. We knew about these problems going into PEP 3156 ( http://python-notes.curiousefficiency.org/en/latest/pep_ideas/async_programming.html#using-special-methods-in-explicitly-asynchronous-code) so it's mainly a matter of having enough experience with asyncio now to be able to suggest specific syntactic sugar to make the right way and the easy way the same way. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Paul, On 2015-04-26 6:25 PM, Paul Sokolovsky wrote: Ok, so here're 3 points this link gives, with my concerns/questions: An alternative idea about new asynchronous iterators and context managers was to reuse existing magic methods, by adding an async keyword to their declarations: [But:] - it would not be possible to create an object that works in both with and async with statements; Yes, and I would say, for good. Behavior of sync and async code is different enough to warrant separate classes (if not libraries!) to implement each paradigm. What if, in otherwise async code, someone will miss async in async with and call sync version of context manager? So, losing ability stated above isn't big practical loss. That's debatable. It might be useful to create hybrid objects that can work in 'with (yield from lock)' and 'async with lock' statements. - it would look confusing Sorry, async def __enter__ doesn't look more confusing than __aenter__ (vs __enter__). I'll update the PEP. The argument shouldn't be that it's confusing, the argument is that __aenter__ returns an 'awaitable', which is either a coroutine-object or a future. You can't reuse __enter__, because you'd break backwards compatibility -- it's perfectly normal for context managers in python to return any object from their __enter__. If we assign some special meaning to futures -- we'll break existing code. and would require some implicit magic behind the scenes in the interpreter; Interpreter already does a lot of implicit magic. Can you please elaborate what exactly would need to be done? Async with implies using YIELD_FROM opcodes. If you want to make regular with statements compatible you'd need to add a lot of opcodes that will at runtime make a decision whether to use YIELD_FROM or not. The other point is that you can't just randomly start using YIELD_FROM, you can only do so from generators/coroutines. one of the main points of this proposal is to make coroutines as simple and foolproof as possible. And it's possible to agree that to separate notions, create a dichotomy is a simple principle on its own. But just taking bunch of stuff - special methods, exceptions - and duplicating it is a bloat a violates another principle - DRY. I'd say that a lot of terrible mistakes has happened in software development precisely because someone followed DRY religiously. Following it here will break backwards compatibility and/or make it harder to understand what is actually going on with your code. You argue that this will make coroutine writing simple. But coroutines are part of the language, and duplicating notions makes language more complex/complicated. Again, it makes reasoning about your code simpler. That what matters. Anyways, I really doubt that you can convince anyone to reuse existing dunder methods for async stuff. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
But new syntax is the whole point of the PEP. I want to be able to *syntactically* tell where the suspension points are in coroutines. Currently this means looking for yield [from]; PEP 492 just adds looking for await and async [for|with]. Making await() a function defeats the purpose because now aliasing can hide its presence, and we're back in the land of gevent or stackless (where *anything* can potentially suspend the current task). I don't want to live in that land. On Sun, Apr 26, 2015 at 1:21 PM, Mark Shannon m...@hotpy.org wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. Please don't add unnecessary new syntax. Cheers, Mark. P.S. I'm not objecting to any of the other new features proposed, just the new syntax. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Paul, On 2015-04-26 8:17 PM, Paul Sokolovsky wrote: Hello, On Sun, 26 Apr 2015 19:45:30 -0400 Yury Selivanov yselivanov...@gmail.com wrote: [] Then, is the only logic for proposing __aenter__ is to reinsure against a situation that someone starts to write async context manager, forgets that they write async context manager, and make an __enter__ method there. It's to make sure that it's impossible to accidentally use existing regular context manager that returns a future object from its __enter__ / __exit__ (nobody prohibits you to return a future object from __exit__, although it's pointless) in an 'async with' block. I see, so it's just to close the final loophole, unlikely to be hit in real life (unless you can say that there're cases of doing just that in existing asyncio libs). Well, given that Greg Ewing wanted even stricter error-proofness, and you rejected it as such strict as to disallow useful behavior, I've just got to trust you that in this case, you're as strict as needed. Well, backwards compatibility is something that better be preserved. Especially with things like context managers. Things with __aiter__/__iter__ are also different. It's easier for everyone to just clearly separate the protocols to avoid all kind of risks. I really don't understand the desire to reuse existing magic methods. Even if it was decided to reuse them, it wouldn't even simplify the implementation in CPython; the code there is already DRY (I don't re-implement opcodes for 'with' statement; I reuse them). Well, there're 3 levels of this stuff: 1. How mere people write their code - everyone would use async def and await, this should be bullet- (and fool-) proof. 2. How library code is written - async iterators won't be written by everyone, and only few will write async context managers; it's fair to expect that people doing these know what they do and don't do stupid mistakes. 3. How it all is coded in particular Python implementation. It's clear that __enter__ vs __aenter__ distinction is 1st kind of issue in your list. It is. As for 3rd point, I'd like to remind that CPython is only one Python implementation. And with my MicroPython hat on, I'd like to know if (some of) these new features are bloat or worthy for the space constraints we have. OT: MicroPython is an amazing project. Kudos for doing it. I really hope that addition of few new magic methods won't make it too hard for you guys to implement PEP 492 in MicroPython one day. Thanks! Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 492: No new syntax is required
Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. Please don't add unnecessary new syntax. Cheers, Mark. P.S. I'm not objecting to any of the other new features proposed, just the new syntax. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
How do you implement async for? On Sun, Apr 26, 2015 at 11:21 PM, Mark Shannon m...@hotpy.org wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. Please don't add unnecessary new syntax. Cheers, Mark. P.S. I'm not objecting to any of the other new features proposed, just the new syntax. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/yoavglazner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
Hi Mark, On 2015-04-26 4:21 PM, Mark Shannon wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Mark, all your points are explained in the PEP in a great detail: Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. Yes, everything that the PEP proposes can be done without new syntax. That's how people use asyncio right now, with only what we have in 3.4. But it's hard. Iterating through something asynchronously? Write a 'while True' loop. Instead of 1 line you now have 5 or 6. Want to commit your database transaction? Instead of 'async with' you will write 'try..except..finally' block, with a very high probability to introduce a bug, because you don't rollback or commit properly or propagate exception. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. https://www.python.org/dev/peps/pep-0492/#rationale-and-goals https://www.python.org/dev/peps/pep-0492/#debugging-features https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? Because you can't reuse them. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-for-and-with-statements https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-magic-names 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. It can't be replaced by a function. Only if you use greenlets or Stackless Python. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. There is no equivalent to 'async with'. with (yield from xxx) only allows you to suspend execution in __enter__ (and it's not actually in __enter__, but in a coroutine that returns a context manager). https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with see New Syntax section to see what 'async with' is equivalent too. Please don't add unnecessary new syntax. It is necessary. Perhaps you haven't spent a lot of time maintaining huge code-bases developed with frameworks like asyncio, so I understand why it does look unnecessary to you. Thanks, Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 492: No new syntax is required
On 26/04/15 21:40, Yury Selivanov wrote: Hi Mark, On 2015-04-26 4:21 PM, Mark Shannon wrote: Hi, I was looking at PEP 492 and it seems to me that no new syntax is required. Mark, all your points are explained in the PEP in a great detail: I did read the PEP. I do think that clarifying the distinction between coroutines and 'normal' generators is a good idea. Adding stuff to the standard library to help is fine. I just don't think that any new syntax is necessary. Looking at the code, it does four things; all of which, or a functional equivalent, could be done with no new syntax. Yes, everything that the PEP proposes can be done without new syntax. That's how people use asyncio right now, with only what we have in 3.4. But it's hard. Iterating through something asynchronously? Write a 'while True' loop. Instead of 1 line you now have 5 or 6. Want to commit your database transaction? Instead of 'async with' you will write 'try..except..finally' block, with a very high probability to introduce a bug, because you don't rollback or commit properly or propagate exception. I don't see why you can't do transactions using a 'with' statement. 1. Make a normal function into a generator or coroutine. This can be done with a decorator. https://www.python.org/dev/peps/pep-0492/#rationale-and-goals states that it is not possible to natively define a coroutine which has no yield or yield from statement which is just not true. https://www.python.org/dev/peps/pep-0492/#debugging-features Requires the addition of the CO_COROUTINE flag, not any new keywords. https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword Seems to be repeating the above. 2. Support a parallel set of special methods starting with 'a' or 'async'. Why not just use the current set of special methods? Because you can't reuse them. https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-for-and-with-statements Which seems back to front. The argument is that existing syntax constructs cannot be made to work with asynchronous objects. Why not make the asynchronous objects work with the existing syntax? https://www.python.org/dev/peps/pep-0492/#why-not-reuse-existing-magic-names The argument here relies on the validity of the previous points. 3. await. await is an operator that takes one argument and produces a single result, without altering flow control and can thus be replaced by an function. It can't be replaced by a function. Only if you use greenlets or Stackless Python. Why not? The implementation of await is here: https://github.com/python/cpython/compare/master...1st1:await#diff-23c87bfada1d01335a3019b9321502a0R642 which clearly could be made into a function. 4. Asynchronous with statement. The PEP lists the equivalent as with (yield from xxx) which doesn't seem so bad. There is no equivalent to 'async with'. with (yield from xxx) only allows you to suspend execution in __enter__ (and it's not actually in __enter__, but in a coroutine that returns a context manager). https://www.python.org/dev/peps/pep-0492/#asynchronous-context-managers-and-async-with see New Syntax section to see what 'async with' is equivalent too. Which, by comparing with PEP 343, can be translated as: with expr as e: e = await(e) ... Please don't add unnecessary new syntax. It is necessary. This isn't an argument, it's just contradiction ;) Perhaps you haven't spent a lot of time maintaining huge code-bases developed with frameworks like asyncio, so I understand why it does look unnecessary to you. This is a good reason for clarifying the distinction between 'normal' generators and coroutines. It is not, IMO, justification for burdening the language (and everyone porting Python 2 code) with extra syntax. Cheers, Mark. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
