New submission from Nick Davies <pyt...@nicolasdavies.com.au>:
This problem was identified in https://bugs.python.org/issue34970 but I think the fix might have been incorrect. The theory in issue34970 was that GC was causing the weakrefset for `all_tasks` to change during iteration. However Weakset provides an `_IterationGuard` class to prevent GC from changing the set during iteration and hence preventing this problem in a single thread. My thoughts on this problem are: - `asyncio.tasks._all_tasks` is shared for all users of asyncio (https://github.com/python/cpython/blob/3.7/Lib/asyncio/tasks.py#L818) - Any new Task constructed mutates `_all_tasks` (https://github.com/python/cpython/blob/3.7/Lib/asyncio/tasks.py#L117) - _IterationGuard won't protect iterations in this case because calls to Weakset.add will always commit changes even if there is something iterating (https://github.com/python/cpython/blob/3.6/Lib/_weakrefset.py#L83) - calls to `asyncio.all_tasks` or `asyncio.tasks.Task.all_tasks` crash if any task is started on any thread during iteration. Repro code: ``` import asyncio from threading import Thread async def do_nothing(): await asyncio.sleep(0) async def loop_tasks(): loop = asyncio.get_event_loop() while True: loop.create_task(do_nothing()) await asyncio.sleep(0.01) def old_thread(): loop = asyncio.new_event_loop() while True: asyncio.tasks.Task.all_tasks(loop=loop) def new_thread(): loop = asyncio.new_event_loop() while True: asyncio.all_tasks(loop=loop) old_t = Thread(target=old_thread) new_t = Thread(target=new_thread) old_t.start() new_t.start() asyncio.run(loop_tasks()) ``` Output: ``` Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "tmp/test_asyncio.py", line 25, in new_thread asyncio.all_tasks(loop=loop) File "/usr/lib/python3.7/asyncio/tasks.py", line 40, in all_tasks return {t for t in list(_all_tasks) File "/usr/lib/python3.7/_weakrefset.py", line 60, in __iter__ for itemref in self.data: RuntimeError: Set changed size during iteration Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/usr/lib/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "tmp/test_asyncio.py", line 19, in old_thread asyncio.tasks.Task.all_tasks(loop=loop) File "/usr/lib/python3.7/asyncio/tasks.py", line 52, in _all_tasks_compat return {t for t in list(_all_tasks) if futures._get_loop(t) is loop} File "/usr/lib/python3.7/_weakrefset.py", line 60, in __iter__ for itemref in self.data: RuntimeError: Set changed size during iteration ``` ---------- components: asyncio messages: 339991 nosy: Nick Davies, asvetlov, yselivanov priority: normal severity: normal status: open title: asyncio.all_tasks() crashes if asyncio is used in multiple threads versions: Python 3.7 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue36607> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
