Andrey Paramonov <para...@acdlabs.ru> added the comment:
Might as_completed() be considered a low-level API, but as of Python 3.7 there are seemingly no ready alternatives to achieve proposed behavior. All of asyncio.gather(), asyncio.wait(), asyncio.as_completed() expect awaitables list of limited size; doing something like https://www.artificialworlds.net/blog/2017/06/12/making-100-million-requests-with-python-aiohttp is not straightforward. A function that takes iterator/async iterator of tasks and is itself generator/async generator is very much wanted, something in the spirit of (but more efficient?) ---- async def igather(tasks, limit=None): pending = set() while True: for task in islice(tasks, limit - len(pending) if limit else None): pending.add(task) if pending: done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED) for task in done: yield task else: break ---- It is an open question whether such function should yield results in the task submission order. Albeit useful, it's a bit harder to implement and (most importantly) has terrible worst-case memory behavior. See also: https://bugs.python.org/issue33533 https://github.com/h2non/paco/issues/38 ---------- nosy: +aparamon _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue30782> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com