Not sure how asyncio can help in this case. It has a warning in debug mode already. Adding `await asyncio.sleep(0)` is the correct fix for your case. I don't think that the code should be a part of asyncio. A recipe is a good idea maybe, not sure. The problem is that the snippet itself is not very helpful. Documentation should say that long-running code in async functions should be avoided. It requires a good clarification of what long-running code is. I'm not a documentation writing expert, especially for such not very obvious areas.
On Fri, Jun 14, 2019 at 1:32 PM Nikita Melentev <[email protected]> wrote: > > At work we faced a problem of long running python code. Our case was a short > task, but huge count of iterations. Something like: > > for x in data_list: > # do 1ms non-io pure python task > > So we block loop for more than 100ms, or even 1000ms. The first naive > solution was "move this to thread" so python will switch thread context and > asyncio loop will not be blocked, but this raised two problems: > * If we do asyncio things in our task (create `Future` in our case), then we > need to pass loop explicitly and use `call_soon_threadsafe` > * We still saw asyncio warnings about blocking the loop. Not sure why, but > maybe because of GIL was not released when main/asyncio thread became active. > > We endend up with wrapper for iterable, which switch «asyncio context» via > `asyncio.sleep(0)` (since we know that sleep(0) have special code, which just > switches context) by time or by count. Here is our code: > > async def iterate_non_blocking(iterable, context_switch_interval=0.01, > context_switch_count=None): > last_context_switch_time = time.perf_counter() > for i, value in enumerate(iterable, start=1): > yield value > switch_context_by_interval = context_switch_interval and \ > (time.perf_counter() - last_context_switch_time) >= > context_switch_interval > switch_context_by_count = context_switch_count and i % > context_switch_count == 0 > if switch_context_by_interval or switch_context_by_count: > await asyncio.sleep(0) > last_context_switch_time = time.perf_counter() > > I'm not sure if this is a good approach, but this solves all our problems for > this kind of blocking cases. Here comes discussable things: > * Is there a better approach for such cases? > * Should this be a part of asyncio? > * Should this be a part of documentation as recipe? > _______________________________________________ > Python-ideas mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/EZWJS6OTBDG7LCX2WECDMPET6I5BMGDU/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Thanks, Andrew Svetlov _______________________________________________ Python-ideas mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/HK43PWOMTSAOOF2ORBFRI46ZFYJYGN5J/ Code of Conduct: http://python.org/psf/codeofconduct/
