On Thu, Apr 23, 2015 at 10:30 AM, Wolfgang Langner <tds333+py...@gmail.com> wrote: > Hi, > > most of the time I am a silent reader but in this discussion I must step in. > I use twisted and async stuff a lot over years followed development of > asyncio closely. > > First it is good to do differentiate async coroutines from generators. So > everyone can see it and have this in mind > and don't mix up booth. It is also easier to explain for new users. > Sometimes generators blows their mind and it takes > a while to get used to them. Async stuff is even harder. > > 1. I am fine with using something special instead of "yield" or "yield from" > for this. C# "await" is ok. > > Everything else suggested complicates the language and makes it harder to > read. > > 2. > async def f(): is harder to read and something special also it breaks the > symmetry in front (def indent). > Also every existing tooling must be changed to support it. Same for def > async, def f() async: > I thing a decorator is enough here > @coroutine > def f(): > is the best solution to mark something as a coroutine. > Sorry, `@coroutine` decorator is part of Python semantics, not Python syntax. That means Python compiler cannot relay on @coroutine in parsing. `away`, `async for` and `async with` are available only inside `async def`, not inside regular `def`. > > 3. > async with and async for > Bead idea, we clutter the language even more and it is one more thing every > newbie could do wrong. > for x in y: > result = await f() > is enough, every 'async' framework lived without it over years.
async for i in iterable: pass is not equal for for fut in iterable: i = yield from fut async for is also suitable when size of iterable sequence is unknown on iteration start. Let's look, for example, on Redis SCAN command (http://redis.io/commands/scan) for iterating over redis keys. It returns bulk of keys and opaque value for next SCAN call. In current Python it must be written as cur = 0 while True: bulk, cur = yield from redis.scan(cur) for key in bulk: process_key(key) if cur == 0: break With new syntax iteration looks much more native: async for key in redis.scan(cur): process_key(key) > Same for with statement. > > The main use case suggested was for database stuff and this is also where > most are best with > defer something to a thread and keep it none async. > `async with` is not only for transactions in relation databases. As well as plain `with` is used not only for databases but for many other things -- from files to decimal contexts. As realistic example not related to databases please recall RabbitMQ. Every message processing may be acknowledged. The native API for that may be while True: async with channel.consume("my queue") as msg: process_msg(msg) with acknowledgement on successful message processing only. Acknowledgement requires network communication and must be asynchronous operation. > > All together it is very late in the development cycle for 3.5 to incorporate > such a big change. > Best is to give all this some more time and defer it to 3.6 and some alpha > releases to experiment with. > > Regards, > > Wolfgang > > > > On Tue, Apr 21, 2015 at 7:26 PM, Yury Selivanov <yselivanov...@gmail.com> > wrote: >> >> Hi python-dev, >> >> I'm moving the discussion from python-ideas to here. >> >> The updated version of the PEP should be available shortly >> at https://www.python.org/dev/peps/pep-0492 >> and is also pasted in this email. >> >> Updates: >> >> 1. CO_ASYNC flag was renamed to CO_COROUTINE; >> >> 2. sys.set_async_wrapper() was renamed to >> sys.set_coroutine_wrapper(); >> >> 3. New function: sys.get_coroutine_wrapper(); >> >> 4. types.async_def() renamed to types.coroutine(); >> >> 5. New section highlighting differences from >> PEP 3152. >> >> 6. New AST node - AsyncFunctionDef; the proposal >> now is 100% backwards compatible; >> >> 7. A new section clarifying that coroutine-generators >> are not part of the current proposal; >> >> 8. Various small edits/typos fixes. >> >> >> There's is a bug tracker issue to track code review >> of the reference implementation (Victor Stinner is >> doing the review): http://bugs.python.org/issue24017 >> While the PEP isn't accepted, we want to make sure >> that the reference implementation is ready when such >> a decision will be made. >> >> >> Let's discuss some open questions: >> >> 1. Victor raised a question if we should locate >> coroutine() function from 'types' module to >> 'functools'. >> >> My opinion is that 'types' module is a better >> place for 'corotine()', since it adjusts the >> type of the passed generator. 'functools' is >> about 'partials', 'lru_cache' and 'wraps' kind >> of things. >> >> 2. I propose to disallow using of 'for..in' loops, >> and builtins like 'list()', 'iter()', 'next()', >> 'tuple()' etc on coroutines. >> >> It's possible by modifying PyObject_GetIter to >> raise an exception if it receives a coroutine-object. >> >> 'yield from' can also be modified to only accept >> coroutine objects if it is called from a generator >> with CO_COROUTINE flag. >> >> This will further separate coroutines from >> generators, making it harder to screw something >> up by an accident. >> >> I have a branch of reference implementation >> https://github.com/1st1/cpython/tree/await_noiter >> where this is implemented. I did not observe >> any performance drop. >> >> There is just one possible backwards compatibility >> issue here: there will be an exception if some user >> of asyncio actually used to iterate over generators >> decorated with @coroutine. But I can't imagine >> why would someone do that, and even if they did -- >> it's probably a bug or wrong usage of asyncio. >> >> >> That's it! I'd be happy to hear some feedback! >> >> Thanks, >> Yury >> >> >> >> PEP: 492 >> Title: Coroutines with async and await syntax >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Yury Selivanov <yseliva...@sprymix.com> >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 09-Apr-2015 >> Python-Version: 3.5 >> Post-History: 17-Apr-2015, 21-Apr-2015 >> >> >> Abstract >> ======== >> >> This PEP introduces new syntax for coroutines, asynchronous ``with`` >> statements and ``for`` loops. The main motivation behind this proposal >> is to streamline writing and maintaining asynchronous code, as well as >> to simplify previously hard to implement code patterns. >> >> >> Rationale and Goals >> =================== >> >> Current Python supports implementing coroutines via generators (PEP >> 342), further enhanced by the ``yield from`` syntax introduced in PEP >> 380. This approach has a number of shortcomings: >> >> * it is easy to confuse coroutines with regular generators, since they >> share the same syntax; async libraries often attempt to alleviate >> this by using decorators (e.g. ``@asyncio.coroutine`` [1]_); >> >> * it is not possible to natively define a coroutine which has no >> ``yield`` or ``yield from`` statements, again requiring the use of >> decorators to fix potential refactoring issues; >> >> * support for asynchronous calls is limited to expressions where >> ``yield`` is allowed syntactically, limiting the usefulness of >> syntactic features, such as ``with`` and ``for`` statements. >> >> This proposal makes coroutines a native Python language feature, and >> clearly separates them from generators. This removes >> generator/coroutine ambiguity, and makes it possible to reliably define >> coroutines without reliance on a specific library. This also enables >> linters and IDEs to improve static code analysis and refactoring. >> >> Native coroutines and the associated new syntax features make it >> possible to define context manager and iteration protocols in >> asynchronous terms. As shown later in this proposal, the new ``async >> with`` statement lets Python programs perform asynchronous calls when >> entering and exiting a runtime context, and the new ``async for`` >> statement makes it possible to perform asynchronous calls in iterators. >> >> >> Specification >> ============= >> >> This proposal introduces new syntax and semantics to enhance coroutine >> support in Python, it does not change the internal implementation of >> coroutines, which are still based on generators. >> >> It is strongly suggested that the reader understands how coroutines are >> implemented in Python (PEP 342 and PEP 380). It is also recommended to >> read PEP 3156 (asyncio framework) and PEP 3152 (Cofunctions). >> >> From this point in this document we use the word *coroutine* to refer >> to functions declared using the new syntax. *generator-based >> coroutine* is used where necessary to refer to coroutines that are >> based on generator syntax. >> >> >> New Coroutine Declaration Syntax >> -------------------------------- >> >> The following new syntax is used to declare a coroutine:: >> >> async def read_data(db): >> pass >> >> Key properties of coroutines: >> >> * ``async def`` functions are always coroutines, even if they do not >> contain ``await`` expressions. >> >> * It is a ``SyntaxError`` to have ``yield`` or ``yield from`` >> expressions in an ``async`` function. >> >> * Internally, a new code object flag - ``CO_COROUTINE`` - is introduced >> to enable runtime detection of coroutines (and migrating existing >> code). All coroutines have both ``CO_COROUTINE`` and ``CO_GENERATOR`` >> flags set. >> >> * Regular generators, when called, return a *generator object*; >> similarly, coroutines return a *coroutine object*. >> >> * ``StopIteration`` exceptions are not propagated out of coroutines, >> and are replaced with a ``RuntimeError``. For regular generators >> such behavior requires a future import (see PEP 479). >> >> >> types.coroutine() >> ----------------- >> >> A new function ``coroutine(gen)`` is added to the ``types`` module. It >> applies ``CO_COROUTINE`` flag to the passed generator-function's code >> object, making it to return a *coroutine object* when called. >> >> This feature enables an easy upgrade path for existing libraries. >> >> >> Await Expression >> ---------------- >> >> The following new ``await`` expression is used to obtain a result of >> coroutine execution:: >> >> async def read_data(db): >> data = await db.fetch('SELECT ...') >> ... >> >> ``await``, similarly to ``yield from``, suspends execution of >> ``read_data`` coroutine until ``db.fetch`` *awaitable* completes and >> returns the result data. >> >> It uses the ``yield from`` implementation with an extra step of >> validating its argument. ``await`` only accepts an *awaitable*, which >> can be one of: >> >> * A *coroutine object* returned from a *coroutine* or a generator >> decorated with ``types.coroutine()``. >> >> * An object with an ``__await__`` method returning an iterator. >> >> Any ``yield from`` chain of calls ends with a ``yield``. This is a >> fundamental mechanism of how *Futures* are implemented. Since, >> internally, coroutines are a special kind of generators, every >> ``await`` is suspended by a ``yield`` somewhere down the chain of >> ``await`` calls (please refer to PEP 3156 for a detailed >> explanation.) >> >> To enable this behavior for coroutines, a new magic method called >> ``__await__`` is added. In asyncio, for instance, to enable Future >> objects in ``await`` statements, the only change is to add >> ``__await__ = __iter__`` line to ``asyncio.Future`` class. >> >> Objects with ``__await__`` method are called *Future-like* objects in >> the rest of this PEP. >> >> Also, please note that ``__aiter__`` method (see its definition >> below) cannot be used for this purpose. It is a different protocol, >> and would be like using ``__iter__`` instead of ``__call__`` for >> regular callables. >> >> It is a ``SyntaxError`` to use ``await`` outside of a coroutine. >> >> >> Asynchronous Context Managers and "async with" >> ---------------------------------------------- >> >> An *asynchronous context manager* is a context manager that is able to >> suspend execution in its *enter* and *exit* methods. >> >> To make this possible, a new protocol for asynchronous context managers >> is proposed. Two new magic methods are added: ``__aenter__`` and >> ``__aexit__``. Both must return an *awaitable*. >> >> An example of an asynchronous context manager:: >> >> class AsyncContextManager: >> async def __aenter__(self): >> await log('entering context') >> >> async def __aexit__(self, exc_type, exc, tb): >> await log('exiting context') >> >> >> New Syntax >> '''''''''' >> >> A new statement for asynchronous context managers is proposed:: >> >> async with EXPR as VAR: >> BLOCK >> >> >> which is semantically equivalent to:: >> >> mgr = (EXPR) >> aexit = type(mgr).__aexit__ >> aenter = type(mgr).__aenter__(mgr) >> exc = True >> >> try: >> try: >> VAR = await aenter >> BLOCK >> except: >> exc = False >> exit_res = await aexit(mgr, *sys.exc_info()) >> if not exit_res: >> raise >> >> finally: >> if exc: >> await aexit(mgr, None, None, None) >> >> >> As with regular ``with`` statements, it is possible to specify multiple >> context managers in a single ``async with`` statement. >> >> It is an error to pass a regular context manager without ``__aenter__`` >> and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError`` >> to use ``async with`` outside of a coroutine. >> >> >> Example >> ''''''' >> >> With asynchronous context managers it is easy to implement proper >> database transaction managers for coroutines:: >> >> async def commit(session, data): >> ... >> >> async with session.transaction(): >> ... >> await session.update(data) >> ... >> >> Code that needs locking also looks lighter:: >> >> async with lock: >> ... >> >> instead of:: >> >> with (yield from lock): >> ... >> >> >> Asynchronous Iterators and "async for" >> -------------------------------------- >> >> An *asynchronous iterable* is able to call asynchronous code in its >> *iter* implementation, and *asynchronous iterator* can call >> asynchronous code in its *next* method. To support asynchronous >> iteration: >> >> 1. An object must implement an ``__aiter__`` method returning an >> *awaitable* resulting in an *asynchronous iterator object*. >> >> 2. An *asynchronous iterator object* must implement an ``__anext__`` >> method returning an *awaitable*. >> >> 3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration`` >> exception. >> >> An example of asynchronous iterable:: >> >> class AsyncIterable: >> async def __aiter__(self): >> return self >> >> async def __anext__(self): >> data = await self.fetch_data() >> if data: >> return data >> else: >> raise StopAsyncIteration >> >> async def fetch_data(self): >> ... >> >> >> New Syntax >> '''''''''' >> >> A new statement for iterating through asynchronous iterators is >> proposed:: >> >> async for TARGET in ITER: >> BLOCK >> else: >> BLOCK2 >> >> which is semantically equivalent to:: >> >> iter = (ITER) >> iter = await type(iter).__aiter__(iter) >> running = True >> while running: >> try: >> TARGET = await type(iter).__anext__(iter) >> except StopAsyncIteration: >> running = False >> else: >> BLOCK >> else: >> BLOCK2 >> >> >> It is an error to pass a regular iterable without ``__aiter__`` method >> to ``async for``. It is a ``SyntaxError`` to use ``async for`` outside >> of a coroutine. >> >> As for with regular ``for`` statement, ``async for`` has an optional >> ``else`` clause. >> >> >> Example 1 >> ''''''''' >> >> With asynchronous iteration protocol it is possible to asynchronously >> buffer data during iteration:: >> >> async for data in cursor: >> ... >> >> Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows >> of data from a database after every ``N`` iterations. >> >> The following code illustrates new asynchronous iteration protocol:: >> >> class Cursor: >> def __init__(self): >> self.buffer = collections.deque() >> >> def _prefetch(self): >> ... >> >> async def __aiter__(self): >> return self >> >> async def __anext__(self): >> if not self.buffer: >> self.buffer = await self._prefetch() >> if not self.buffer: >> raise StopAsyncIteration >> return self.buffer.popleft() >> >> then the ``Cursor`` class can be used as follows:: >> >> async for row in Cursor(): >> print(row) >> >> which would be equivalent to the following code:: >> >> i = await Cursor().__aiter__() >> while True: >> try: >> row = await i.__anext__() >> except StopAsyncIteration: >> break >> else: >> print(row) >> >> >> Example 2 >> ''''''''' >> >> The following is a utility class that transforms a regular iterable to >> an asynchronous one. While this is not a very useful thing to do, the >> code illustrates the relationship between regular and asynchronous >> iterators. >> >> :: >> >> class AsyncIteratorWrapper: >> def __init__(self, obj): >> self._it = iter(obj) >> >> async def __aiter__(self): >> return self >> >> async def __anext__(self): >> try: >> value = next(self._it) >> except StopIteration: >> raise StopAsyncIteration >> return value >> >> async for item in AsyncIteratorWrapper("abc"): >> print(item) >> >> >> Why StopAsyncIteration? >> ''''''''''''''''''''''' >> >> Coroutines are still based on generators internally. So, before PEP >> 479, there was no fundamental difference between >> >> :: >> >> def g1(): >> yield from fut >> return 'spam' >> >> and >> >> :: >> >> def g2(): >> yield from fut >> raise StopIteration('spam') >> >> And since PEP 479 is accepted and enabled by default for coroutines, >> the following example will have its ``StopIteration`` wrapped into a >> ``RuntimeError`` >> >> :: >> >> async def a1(): >> await fut >> raise StopIteration('spam') >> >> The only way to tell the outside code that the iteration has ended is >> to raise something other than ``StopIteration``. Therefore, a new >> built-in exception class ``StopAsyncIteration`` was added. >> >> Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions >> raised in coroutines are wrapped in ``RuntimeError``. >> >> >> Debugging Features >> ------------------ >> >> One of the most frequent mistakes that people make when using >> generators as coroutines is forgetting to use ``yield from``:: >> >> @asyncio.coroutine >> def useful(): >> asyncio.sleep(1) # this will do noting without 'yield from' >> >> For debugging this kind of mistakes there is a special debug mode in >> asyncio, in which ``@coroutine`` decorator wraps all functions with a >> special object with a destructor logging a warning. Whenever a wrapped >> generator gets garbage collected, a detailed logging message is >> generated with information about where exactly the decorator function >> was defined, stack trace of where it was collected, etc. Wrapper >> object also provides a convenient ``__repr__`` function with detailed >> information about the generator. >> >> The only problem is how to enable these debug capabilities. Since >> debug facilities should be a no-op in production mode, ``@coroutine`` >> decorator makes the decision of whether to wrap or not to wrap based on >> an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is >> possible to run asyncio programs with asyncio's own functions >> instrumented. ``EventLoop.set_debug``, a different debug facility, has >> no impact on ``@coroutine`` decorator's behavior. >> >> With this proposal, coroutines is a native, distinct from generators, >> concept. New methods ``set_coroutine_wrapper`` and >> ``get_coroutine_wrapper`` are added to the ``sys`` module, with which >> frameworks can provide advanced debugging facilities. >> >> It is also important to make coroutines as fast and efficient as >> possible, therefore there are no debug features enabled by default. >> >> Example:: >> >> async def debug_me(): >> await asyncio.sleep(1) >> >> def async_debug_wrap(generator): >> return asyncio.AsyncDebugWrapper(generator) >> >> sys.set_coroutine_wrapper(async_debug_wrap) >> >> debug_me() # <- this line will likely GC the coroutine object and >> # trigger AsyncDebugWrapper's code. >> >> assert isinstance(debug_me(), AsyncDebugWrapper) >> >> sys.set_coroutine_wrapper(None) # <- this unsets any >> # previously set wrapper >> assert not isinstance(debug_me(), AsyncDebugWrapper) >> >> If ``sys.set_coroutine_wrapper()`` is called twice, the new wrapper >> replaces the previous wrapper. ``sys.set_coroutine_wrapper(None)`` >> unsets the wrapper. >> >> >> Glossary >> ======== >> >> :Coroutine: >> A coroutine function, or just "coroutine", is declared with ``async >> def``. It uses ``await`` and ``return value``; see `New Coroutine >> Declaration Syntax`_ for details. >> >> :Coroutine object: >> Returned from a coroutine function. See `Await Expression`_ for >> details. >> >> :Future-like object: >> An object with an ``__await__`` method. Can be consumed by an >> ``await`` expression in a coroutine. A coroutine waiting for a >> Future-like object is suspended until the Future-like object's >> ``__await__`` completes, and returns the result. See `Await >> Expression`_ for details. >> >> :Awaitable: >> A *Future-like* object or a *coroutine object*. See `Await >> Expression`_ for details. >> >> :Generator-based coroutine: >> Coroutines based in generator syntax. Most common example is >> ``@asyncio.coroutine``. >> >> :Asynchronous context manager: >> An asynchronous context manager has ``__aenter__`` and ``__aexit__`` >> methods and can be used with ``async with``. See `Asynchronous >> Context Managers and "async with"`_ for details. >> >> :Asynchronous iterable: >> An object with an ``__aiter__`` method, which must return an >> *asynchronous iterator* object. Can be used with ``async for``. >> See `Asynchronous Iterators and "async for"`_ for details. >> >> :Asynchronous iterator: >> An asynchronous iterator has an ``__anext__`` method. See >> `Asynchronous Iterators and "async for"`_ for details. >> >> >> List of functions and methods >> ============================= >> >> ================= =================================== ================= >> Method Can contain Can't contain >> ================= =================================== ================= >> async def func await, return value yield, yield from >> async def __a*__ await, return value yield, yield from >> def __a*__ return awaitable await >> def __await__ yield, yield from, return iterable await >> generator yield, yield from, return value await >> ================= =================================== ================= >> >> Where: >> >> * "async def func": coroutine; >> >> * "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, >> ``__aexit__`` defined with the ``async`` keyword; >> >> * "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``, >> ``__aexit__`` defined without the ``async`` keyword, must return an >> *awaitable*; >> >> * "def __await__": ``__await__`` method to implement *Future-like* >> objects; >> >> * generator: a "regular" generator, function defined with ``def`` and >> which contains a least one ``yield`` or ``yield from`` expression. >> >> >> Transition Plan >> =============== >> >> To avoid backwards compatibility issues with ``async`` and ``await`` >> keywords, it was decided to modify ``tokenizer.c`` in such a way, that >> it: >> >> * recognizes ``async def`` name tokens combination (start of a >> coroutine); >> >> * keeps track of regular functions and coroutines; >> >> * replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with >> ``AWAIT`` when in the process of yielding tokens for coroutines. >> >> This approach allows for seamless combination of new syntax features >> (all of them available only in ``async`` functions) with any existing >> code. >> >> An example of having "async def" and "async" attribute in one piece of >> code:: >> >> class Spam: >> async = 42 >> >> async def ham(): >> print(getattr(Spam, 'async')) >> >> # The coroutine can be executed and will print '42' >> >> >> Backwards Compatibility >> ----------------------- >> >> This proposal preserves 100% backwards compatibility. >> >> >> Grammar Updates >> --------------- >> >> Grammar changes are also fairly minimal:: >> >> await_expr: AWAIT test >> await_stmt: await_expr >> >> decorated: decorators (classdef | funcdef | async_funcdef) >> async_funcdef: ASYNC funcdef >> >> async_stmt: ASYNC (funcdef | with_stmt | for_stmt) >> >> compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | >> with_stmt | funcdef | classdef | decorated | >> async_stmt) >> >> atom: ('(' [yield_expr|await_expr|testlist_comp] ')' | >> '[' [testlist_comp] ']' | >> '{' [dictorsetmaker] '}' | >> NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False’) >> >> expr_stmt: testlist_star_expr >> (augassign (yield_expr|await_expr|testlist) | >> ('=' (yield_expr|await_expr|testlist_star_expr))*) >> >> >> Transition Period Shortcomings >> ------------------------------ >> >> There is just one. >> >> Until ``async`` and ``await`` are not proper keywords, it is not >> possible (or at least very hard) to fix ``tokenizer.c`` to recognize >> them on the **same line** with ``def`` keyword:: >> >> # async and await will always be parsed as variables >> >> async def outer(): # 1 >> def nested(a=(await fut)): >> pass >> >> async def foo(): return (await fut) # 2 >> >> Since ``await`` and ``async`` in such cases are parsed as ``NAME`` >> tokens, a ``SyntaxError`` will be raised. >> >> To workaround these issues, the above examples can be easily rewritten >> to a more readable form:: >> >> async def outer(): # 1 >> a_default = await fut >> def nested(a=a_default): >> pass >> >> async def foo(): # 2 >> return (await fut) >> >> This limitation will go away as soon as ``async`` and ``await`` ate >> proper keywords. Or if it's decided to use a future import for this >> PEP. >> >> >> Deprecation Plans >> ----------------- >> >> ``async`` and ``await`` names will be softly deprecated in CPython 3.5 >> and 3.6. In 3.7 we will transform them to proper keywords. Making >> ``async`` and ``await`` proper keywords before 3.7 might make it harder >> for people to port their code to Python 3. >> >> >> asyncio >> ------- >> >> ``asyncio`` module was adapted and tested to work with coroutines and >> new statements. Backwards compatibility is 100% preserved. >> >> The required changes are mainly: >> >> 1. Modify ``@asyncio.coroutine`` decorator to use new >> ``types.coroutine()`` function. >> >> 2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class. >> >> 3. Add ``ensure_task()`` as an alias for ``async()`` function. >> Deprecate ``async()`` function. >> >> >> Design Considerations >> ===================== >> >> PEP 3152 >> -------- >> >> PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines >> (called "cofunctions"). Some key points: >> >> 1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is >> always a generator, even if there is no ``cocall`` expressions >> inside it. Maps to ``async def`` in this proposal. >> >> 2. A new keyword ``cocall`` to call a *cofunction*. Can only be used >> inside a *cofunction*. Maps to ``await`` in this proposal (with >> some differences, see below.) >> >> 3. It is not possible to call a *cofunction* without a ``cocall`` >> keyword. >> >> 4. ``cocall`` grammatically requires parentheses after it:: >> >> atom: cocall | <existing alternatives for atom> >> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' >> cotrailer: '[' subscriptlist ']' | '.' NAME >> >> 5. ``cocall f(*args, **kwds)`` is semantically equivalent to >> ``yield from f.__cocall__(*args, **kwds)``. >> >> Differences from this proposal: >> >> 1. There is no equivalent of ``__cocall__`` in this PEP, which is >> called and its result is passed to ``yield from`` in the ``cocall`` >> expression. ``await`` keyword expects an *awaitable* object, >> validates the type, and executes ``yield from`` on it. Although, >> ``__await__`` method is similar to ``__cocall__``, but is only used >> to define *Future-like* objects. >> >> 2. ``await`` is defined in almost the same way as ``yield from`` in the >> grammar (it is later enforced that ``await`` can only be inside >> ``async def``). It is possible to simply write ``await future``, >> whereas ``cocall`` always requires parentheses. >> >> 3. To make asyncio work with PEP 3152 it would be required to modify >> ``@asyncio.coroutine`` decorator to wrap all functions in an object >> with a ``__cocall__`` method. To call *cofunctions* from existing >> generator-based coroutines it would be required to use ``costart`` >> built-in. In this proposal ``@asyncio.coroutine`` simply sets >> ``CO_COROUTINE`` on the wrapped function's code object and >> everything works automatically. >> >> 4. Since it is impossible to call a *cofunction* without a ``cocall`` >> keyword, it automatically prevents the common mistake of forgetting >> to use ``yield from`` on generator-based coroutines. This proposal >> addresses this problem with a different approach, see `Debugging >> Features`_. >> >> 5. A shortcoming of requiring a ``cocall`` keyword to call a coroutine >> is that if is decided to implement coroutine-generators -- >> coroutines with ``yield`` or ``async yield`` expressions -- we >> wouldn't need a ``cocall`` keyword to call them. So we'll end up >> having ``__cocall__`` and no ``__call__`` for regular coroutines, >> and having ``__call__`` and no ``__cocall__`` for coroutine- >> generators. >> >> 6. There are no equivalents of ``async for`` and ``async with`` in PEP >> 3152. >> >> >> Coroutine-generators >> -------------------- >> >> With ``async for`` keyword it is desirable to have a concept of a >> *coroutine-generator* -- a coroutine with ``yield`` and ``yield from`` >> expressions. To avoid any ambiguity with regular generators, we would >> likely require to have an ``async`` keyword before ``yield``, and >> ``async yield from`` would raise a ``StopAsyncIteration`` exception. >> >> While it is possible to implement coroutine-generators, we believe that >> they are out of scope of this proposal. It is an advanced concept that >> should be carefully considered and balanced, with a non-trivial changes >> in the implementation of current generator objects. This is a matter >> for a separate PEP. >> >> >> No implicit wrapping in Futures >> ------------------------------- >> >> There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A >> key difference is that JavaScript "async functions" always return a >> Promise. While this approach has some advantages, it also implies that >> a new Promise object is created on each "async function" invocation. >> >> We could implement a similar functionality in Python, by wrapping all >> coroutines in a Future object, but this has the following >> disadvantages: >> >> 1. Performance. A new Future object would be instantiated on each >> coroutine call. Moreover, this makes implementation of ``await`` >> expressions slower (disabling optimizations of ``yield from``). >> >> 2. A new built-in ``Future`` object would need to be added. >> >> 3. Coming up with a generic ``Future`` interface that is usable for any >> use case in any framework is a very hard to solve problem. >> >> 4. It is not a feature that is used frequently, when most of the code >> is coroutines. >> >> >> Why "async" and "await" keywords >> -------------------------------- >> >> async/await is not a new concept in programming languages: >> >> * C# has it since long time ago [5]_; >> >> * proposal to add async/await in ECMAScript 7 [2]_; >> see also Traceur project [9]_; >> >> * Facebook's Hack/HHVM [6]_; >> >> * Google's Dart language [7]_; >> >> * Scala [8]_; >> >> * proposal to add async/await to C++ [10]_; >> >> * and many other less popular languages. >> >> This is a huge benefit, as some users already have experience with >> async/await, and because it makes working with many languages in one >> project easier (Python with ECMAScript 7 for instance). >> >> >> Why "__aiter__" is a coroutine >> ------------------------------ >> >> In principle, ``__aiter__`` could be a regular function. There are >> several good reasons to make it a coroutine: >> >> * as most of the ``__anext__``, ``__aenter__``, and ``__aexit__`` >> methods are coroutines, users would often make a mistake defining it >> as ``async`` anyways; >> >> * there might be a need to run some asynchronous operations in >> ``__aiter__``, for instance to prepare DB queries or do some file >> operation. >> >> >> Importance of "async" keyword >> ----------------------------- >> >> While it is possible to just implement ``await`` expression and treat >> all functions with at least one ``await`` as coroutines, this approach >> makes APIs design, code refactoring and its long time support harder. >> >> Let's pretend that Python only has ``await`` keyword:: >> >> def useful(): >> ... >> await log(...) >> ... >> >> def important(): >> await useful() >> >> If ``useful()`` function is refactored and someone removes all >> ``await`` expressions from it, it would become a regular python >> function, and all code that depends on it, including ``important()`` >> would be broken. To mitigate this issue a decorator similar to >> ``@asyncio.coroutine`` has to be introduced. >> >> >> Why "async def" >> --------------- >> >> For some people bare ``async name(): pass`` syntax might look more >> appealing than ``async def name(): pass``. It is certainly easier to >> type. But on the other hand, it breaks the symmetry between ``async >> def``, ``async with`` and ``async for``, where ``async`` is a modifier, >> stating that the statement is asynchronous. It is also more consistent >> with the existing grammar. >> >> >> Why not a __future__ import >> --------------------------- >> >> ``__future__`` imports are inconvenient and easy to forget to add. >> Also, they are enabled for the whole source file. Consider that there >> is a big project with a popular module named "async.py". With future >> imports it is required to either import it using ``__import__()`` or >> ``importlib.import_module()`` calls, or to rename the module. The >> proposed approach makes it possible to continue using old code and >> modules without a hassle, while coming up with a migration plan for >> future python versions. >> >> >> Why magic methods start with "a" >> -------------------------------- >> >> New asynchronous magic methods ``__aiter__``, ``__anext__``, >> ``__aenter__``, and ``__aexit__`` all start with the same prefix "a". >> An alternative proposal is to use "async" prefix, so that ``__aiter__`` >> becomes ``__async_iter__``. However, to align new magic methods with >> the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided >> to use a shorter version. >> >> >> Why not reuse existing magic names >> ---------------------------------- >> >> An alternative idea about new asynchronous iterators and context >> managers was to reuse existing magic methods, by adding an ``async`` >> keyword to their declarations:: >> >> class CM: >> async def __enter__(self): # instead of __aenter__ >> ... >> >> This approach has the following downsides: >> >> * it would not be possible to create an object that works in both >> ``with`` and ``async with`` statements; >> >> * it would look confusing and would require some implicit magic behind >> the scenes in the interpreter; >> >> * one of the main points of this proposal is to make coroutines as >> simple and foolproof as possible. >> >> >> Comprehensions >> -------------- >> >> For the sake of restricting the broadness of this PEP there is no new >> syntax for asynchronous comprehensions. This should be considered in a >> separate PEP, if there is a strong demand for this feature. >> >> >> Async lambdas >> ------------- >> >> Lambda coroutines are not part of this proposal. In this proposal they >> would look like ``async lambda(parameters): expression``. Unless there >> is a strong demand to have them as part of this proposal, it is >> recommended to consider them later in a separate PEP. >> >> >> Performance >> =========== >> >> Overall Impact >> -------------- >> >> This proposal introduces no observable performance impact. Here is an >> output of python's official set of benchmarks [4]_: >> >> :: >> >> python perf.py -r -b default ../cpython/python.exe >> ../cpython-aw/python.exe >> >> [skipped] >> >> Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0: >> Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 >> x86_64 i386 >> >> Total CPU cores: 8 >> >> ### etree_iterparse ### >> Min: 0.365359 -> 0.349168: 1.05x faster >> Avg: 0.396924 -> 0.379735: 1.05x faster >> Significant (t=9.71) >> Stddev: 0.01225 -> 0.01277: 1.0423x larger >> >> The following not significant results are hidden, use -v to show them: >> django_v2, 2to3, etree_generate, etree_parse, etree_process, >> fastpickle, >> fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http. >> >> >> Tokenizer modifications >> ----------------------- >> >> There is no observable slowdown of parsing python files with the >> modified tokenizer: parsing of one 12Mb file >> (``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount >> of time. >> >> >> async/await >> ----------- >> >> The following micro-benchmark was used to determine performance >> difference between "async" functions and generators:: >> >> import sys >> import time >> >> def binary(n): >> if n <= 0: >> return 1 >> l = yield from binary(n - 1) >> r = yield from binary(n - 1) >> return l + 1 + r >> >> async def abinary(n): >> if n <= 0: >> return 1 >> l = await abinary(n - 1) >> r = await abinary(n - 1) >> return l + 1 + r >> >> def timeit(gen, depth, repeat): >> t0 = time.time() >> for _ in range(repeat): >> list(gen(depth)) >> t1 = time.time() >> print('{}({}) * {}: total {:.3f}s'.format( >> gen.__name__, depth, repeat, t1-t0)) >> >> The result is that there is no observable performance difference. >> Minimum timing of 3 runs >> >> :: >> >> abinary(19) * 30: total 12.985s >> binary(19) * 30: total 12.953s >> >> Note that depth of 19 means 1,048,575 calls. >> >> >> Reference Implementation >> ======================== >> >> The reference implementation can be found here: [3]_. >> >> List of high-level changes and new protocols >> -------------------------------------------- >> >> 1. New syntax for defining coroutines: ``async def`` and new ``await`` >> keyword. >> >> 2. New ``__await__`` method for Future-like objects. >> >> 3. New syntax for asynchronous context managers: ``async with``. And >> associated protocol with ``__aenter__`` and ``__aexit__`` methods. >> >> 4. New syntax for asynchronous iteration: ``async for``. And >> associated protocol with ``__aiter__``, ``__aexit__`` and new built- >> in exception ``StopAsyncIteration``. >> >> 5. New AST nodes: ``AsyncFunctionDef``, ``AsyncFor``, ``AsyncWith``, >> ``Await``. >> >> 6. New functions: ``sys.set_coroutine_wrapper(callback)``, >> ``sys.get_coroutine_wrapper()``, and ``types.coroutine(gen)``. >> >> 7. New ``CO_COROUTINE`` bit flag for code objects. >> >> While the list of changes and new things is not short, it is important >> to understand, that most users will not use these features directly. >> It is intended to be used in frameworks and libraries to provide users >> with convenient to use and unambiguous APIs with ``async def``, >> ``await``, ``async for`` and ``async with`` syntax. >> >> >> Working example >> --------------- >> >> All concepts proposed in this PEP are implemented [3]_ and can be >> tested. >> >> :: >> >> import asyncio >> >> async def echo_server(): >> print('Serving on localhost:8000') >> await asyncio.start_server(handle_connection, >> 'localhost', 8000) >> >> async def handle_connection(reader, writer): >> print('New connection...') >> >> while True: >> data = await reader.read(8192) >> >> if not data: >> break >> >> print('Sending {:.10}... back'.format(repr(data))) >> writer.write(data) >> >> loop = asyncio.get_event_loop() >> loop.run_until_complete(echo_server()) >> try: >> loop.run_forever() >> finally: >> loop.close() >> >> >> References >> ========== >> >> .. [1] >> https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine >> >> .. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions >> >> .. [3] https://github.com/1st1/cpython/tree/await >> >> .. [4] https://hg.python.org/benchmarks >> >> .. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx >> >> .. [6] http://docs.hhvm.com/manual/en/hack.async.php >> >> .. [7] https://www.dartlang.org/articles/await-async/ >> >> .. [8] http://docs.scala-lang.org/sips/pending/async.html >> >> .. [9] >> https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-functions-experimental >> >> .. [10] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf >> (PDF) >> >> >> Acknowledgments >> =============== >> >> I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew >> Svetlov, and Łukasz Langa for their initial feedback. >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/tds333%2Bpydev%40gmail.com > > > > > -- > bye by Wolfgang > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com > -- Thanks, Andrew Svetlov _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com