New submission from Yury Selivanov <yseliva...@gmail.com>:
I believe I might have discovered a problem with asynchronous generators in 3.8. # Prelude In Python prior to 3.8 it was possible to overlap running of "asend()" and "athrow()" methods for the same asynchronous generator. In plain English, it was possible to await on "agen.asend()" while some other coroutine is already awaiting on "agen.asend()". That created all kinds of problems with asynchronous generators when people used them to implement channels by trying to push values into the same asynchronous generator from different coroutines. Regular code that used asynchronous generators in plain and non-sophisticated way did not care. For classic synchronous generators we have a check for preventing overlapping use of "send()" and "throw()" -- the "gi_running" flag. If the flag is set, both methods raise a RuntimeError saying that "generator already executing". # Python 3.8 As was discussed in issues #30773 and #32526 we decided to replicate the same behavior for asynchronous generators, mainly: * add an "ag_running" flag; * set "ag_running" when "anext()" or "athrow()" begin to rung, and set it off when they are finished executing; * if the flag is set when we are about to run "anext()" or "athrow()" it means that another coroutine is reusing the same generator object in parallel and so we raise a RuntimeError. # Problem Closing a generator involves throwing a GeneratorExit exception into it. Throwing the exception is done via calling "throw()" for sync generators, and "athrow()" for async generators. As shown in https://gist.github.com/1st1/d9860cbf6fe2e5d243e695809aea674c, it's an error to close a synchronous generator while it is being iterated. This is how async generators *started to behave* in 3.8. The problem is that asyncio's "loop.shutdown_asyncgens()" method tries to shutdown orphan asynchronous generators by calling "aclose()" on them. The method is public API and is called by "asyncio.run()" automatically. Prior to 3.8, calling "aclose()" worked (maybe not in the most clean way). A GeneratorExit was thrown into an asynchronous generator regardless of whether it was running or not, aborting the execution. In 3.8, calling "aclose()" can crash with a RuntimeError. It's no longer possible to *reliably cancel* a running asynchrounous generator. # Dilemma Replicating the behavior of synchronous generators in asynchronous generators seems like the right thing. But it seems that the requirements for asynchronous generators are different, and 3.8 breaks backwards compat. # Proposed Solution We keep "asend()" and "athrow()" as is in 3.8. They will continue to raise RuntimeError if used in parallel on the same async generator. We modify "aclose()" to allow it being called in parallel with "asend()" or "athrow()". This will restore the <3.8 behavior and fix the "loop.shutdown_asyncgens()" method. Thoughts? ---------- components: Interpreter Core messages: 355158 nosy: asvetlov, gvanrossum, lukasz.langa, ncoghlan, njs, yselivanov priority: normal severity: normal status: open title: async generators aclose() behavior in 3.8 type: behavior versions: Python 3.8, Python 3.9 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue38559> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
