[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: Now that the `versionadded` label has been added and the contextvar issue was addressed, this issue can be closed. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue32309] Implement asyncio.run_in_executor shortcut

miss-islington added the comment: New changeset 3e650545bfe949fa435b0d41e54986f615891ec8 by Miss Islington (bot) in branch '3.9': bpo-32309: Add support for contextvars in asyncio.to_thread() (GH-20278) https://github.com/python/cpython/commit/3e650545bfe949fa435b0d41e54986f615891ec8

[issue32309] Implement asyncio.run_in_executor shortcut

miss-islington added the comment: New changeset 0f56263e62ba91d0baae40fb98947a3a98034a73 by Kyle Stanley in branch 'master': bpo-32309: Add support for contextvars in asyncio.to_thread() (GH-20278) https://github.com/python/cpython/commit/0f56263e62ba91d0baae40fb98947a3a98034a73 --

[issue32309] Implement asyncio.run_in_executor shortcut

Change by miss-islington : -- pull_requests: +19556 pull_request: https://github.com/python/cpython/pull/20279 ___ Python tracker ___

[issue32309] Implement asyncio.run_in_executor shortcut

Change by Kyle Stanley : -- pull_requests: +19555 pull_request: https://github.com/python/cpython/pull/20278 ___ Python tracker ___

[issue32309] Implement asyncio.run_in_executor shortcut

STINNER Victor added the comment: Note for myself: Python 3.9 release manager (Lukasz) approved adding the feature to Python 3.9.0 beta2: https://github.com/python/cpython/pull/20212#pullrequestreview-414278938 -- nosy: +vstinner versions: +Python 3.9 -Python 3.7

[issue32309] Implement asyncio.run_in_executor shortcut

miss-islington added the comment: New changeset e2991308c9b49547d9762157ac913dda94b5eb32 by Miss Islington (bot) in branch '3.9': bpo-32309: Implement asyncio.to_thread() (GH-20143) https://github.com/python/cpython/commit/e2991308c9b49547d9762157ac913dda94b5eb32 --

[issue32309] Implement asyncio.run_in_executor shortcut

Change by miss-islington : -- pull_requests: +19508 pull_request: https://github.com/python/cpython/pull/20212 ___ Python tracker ___

[issue32309] Implement asyncio.run_in_executor shortcut

miss-islington added the comment: New changeset cc2bbc2227c3f5ed9d8f6b3bd052e6f9e68279d2 by Kyle Stanley in branch 'master': bpo-32309: Implement asyncio.to_thread() (GH-20143) https://github.com/python/cpython/commit/cc2bbc2227c3f5ed9d8f6b3bd052e6f9e68279d2 -- nosy:

[issue32309] Implement asyncio.run_in_executor shortcut

Change by Kyle Stanley : -- pull_requests: +19448 pull_request: https://github.com/python/cpython/pull/20143 ___ Python tracker ___

[issue32309] Implement asyncio.run_in_executor shortcut

Change by Chris Jerdonek : -- nosy: +chris.jerdonek ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue32309] Implement asyncio.run_in_executor shortcut

Change by Kyle Stanley : -- pull_requests: +17784 pull_request: https://github.com/python/cpython/pull/18410 ___ Python tracker ___

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > So, I just had an interesting idea... what if ThreadPool.run() returned a > Task instead of a coroutine object? After having some time to think this over, I prefer the current behavior. I don't think there would be significant enough improvement from

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: So, I just had an interesting idea... what if ThreadPool.run() returned a Task instead of a coroutine object? With the current version of asyncio.ThreadPool, if a user wants to create a Task, they would have to do something like this: async with

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > (a) design the API correctly; > (b) ship something that definitely works with a proven ThreadPoolExecutor; Yury and Andrew, here's my latest API design for asyncio.ThreadPool: https://github.com/python/cpython/compare/master...aeros:asyncio-threadpool.

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > (a) design the API correctly; (b) ship something that definitely works with a proven ThreadPoolExecutor; (c) write lots of tests; (d) write docs; (e) if (a-d) are OK, refine the implementation later by replacing ThreadPoolExecutor with a proper (eager

[issue32309] Implement asyncio.run_in_executor shortcut

Yury Selivanov added the comment: > I think that I'm still not understanding something important though. Even if > we initialize our ThreadPoolExecutor outside of __init__ (in a start() > coroutine method, as your proposing), it seems like the threads will be > spawned throughout the

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > since the new threads are spawned in ThreadPoolExecutor *after* > executor.submit() is called It's also worth mentioning that ThreadPoolExecutor only spawns up to one additional thread at a time for each executor.submit() called. --

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > Nice work! This is a great excercise, but we can really just use > concurrent.futures.ThreadPool as is. Spawning threads is fast. As I mentioned > before all we need to do is to design *our* API to NOT initialize pools in > __init__, that's it. The design

[issue32309] Implement asyncio.run_in_executor shortcut

Yury Selivanov added the comment: > I'm going to have to rescind the above statements. I was able to implement a > new prototype of asyncio.ThreadPool (using ThreadPoolExecutor) that spawns > it's threads asynchronously on startup. Since this one a bit more involved > than the previous code

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > The asynchronous spawning of threads or processes would also not be > compatible with the executor subclasses as far as I can tell. > Thus, it seemed to make more sense to me to actually build up a new Pool > class from scratch that was largely based on

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > No, that would be too much work. Writing a thread pool or process pool from > scratch is an extremely tedious task and it will take us years to stabilize > the code. It's not simple. > We should design *our* API correctly though. And that means that we

[issue32309] Implement asyncio.run_in_executor shortcut

Yury Selivanov added the comment: > From my understanding, the executor classes are designed around spawning the > threads (or processes in the case of ProcessPoolExecutor) as needed up to > max_workers, rather than spawning them upon startup. The asynchronous > spawning of threads or

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > And that's why I like it. If we add ProcessPool it will have the same > argument: concurrency. > max_workers isn't correct, as we want to spawn all threads and all processes > when we start. Thus btw makes me think that initializing threads/processes in >

[issue32309] Implement asyncio.run_in_executor shortcut

Yury Selivanov added the comment: > > > > IMO, I think it would be a bit more clear to just explicitly call it > "threads" or "max_threads", as that explains what it's effectively doing. > While "concurrency" is still a perfectly correct way of describing the > behavior, I think it might be

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > def __init__(self, concurrency=None): Minor clarification: the default should probably be None, which would effectively set the default maximum number of threads to min(32, os.cpu_count() + 4), once it's passed to ThreadPoolExecutor. --

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > Number of OS threads to spawn. Ah I see, so this would correspond with the "max_workers" argument of ThreadPoolExecutor then, correct? If so, we could pass this in the __init__ for ThreadPool: def __init__(self, concurrency): ... self._executor =

[issue32309] Implement asyncio.run_in_executor shortcut

Yury Selivanov added the comment: >> async with asyncio.ThreadPool(concurrency=10) as pool: > I'm definitely on board with the usage of an async context manager and the > functionality shown in the example, but I'm not sure that I entirely > understand what the "concurrency" kwarg in

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > I believe behavior occurs within shutdown_default_executor(), correct? > Specifically, within for ThreadPoolExecutor when executor.shutdown(wait=True) > is called and all of the threads are joined without a timeout, it simply > waits for each thread to

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > async with asyncio.ThreadPool(concurrency=10) as pool: I'm definitely on board with the usage of an async context manager and the functionality shown in the example, but I'm not sure that I entirely understand what the "concurrency" kwarg in

[issue32309] Implement asyncio.run_in_executor shortcut

Yury Selivanov added the comment: Few thoughts: 1. I like the idea of having a context manager to create a thread pool. It should be initialized in a top-level coroutine and the passed down to other code, as in: async def main(): async with asyncio.ThreadPool(concurrency=10) as pool:

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > Also in this case run awaits and returns the result. Yury suggested earlier > just to return the future and not await. Yeah that's roughly what my initial version was doing. I'm personally leaning a bit more towards returning the future rather than the

[issue32309] Implement asyncio.run_in_executor shortcut

Paul Martin added the comment: Good points. I made a mistake in run Should be: async def run(self, func, *args, **kwargs): call = functools.partial(func, *args, **kwargs) return await self._loop.run_in_executor(self._executor, call) Also in this case run awaits and

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: Actually, I think it would be better to move the functionality of loop.shutdown_default_executor() to a new private method loop._shutdown_executor() that takes an executor argument rather than shutting down the default one. This could be used in both

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > thread = threading.Thread(target=loop._do_shutdown, args=(executor,future)) Correction: > thread = threading.Thread(target=_do_shutdown, args=(loop, executor,future)) Also, it might make more sense to rename _do_shutdown() to _do_executor_shutdown() to

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: Also, I agree with Paul's idea of initializing the ThreadPoolExecutor in the __init__ instead of __aenter__, that makes more sense now that I think about it. -- ___ Python tracker

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > I don't think changing the default executor is a good approach. What happens, > if two or more thread pools are running at the same time? In that case they > will use the same default executor anyway, so creating a new executor each > time seems like a

[issue32309] Implement asyncio.run_in_executor shortcut

Andrew Svetlov added the comment: Paul's version looks better. Two notes: 1. get_running_loop() should be used instead of get_event_loop() 2. There is no `await executer.shutdown()` API, the method is synchronous. -- ___ Python tracker

[issue32309] Implement asyncio.run_in_executor shortcut

Paul Martin added the comment: Run method should be: async def run(self, func, *args, **kwargs): call = functools.partial(func, *args, **kwargs) return await self._loop.run_in_executor(None, call) -- ___ Python tracker

[issue32309] Implement asyncio.run_in_executor shortcut

Paul Martin added the comment: I don't think changing the default executor is a good approach. What happens, if two or more thread pools are running at the same time? In that case they will use the same default executor anyway, so creating a new executor each time seems like a waste.

[issue32309] Implement asyncio.run_in_executor shortcut

Andrew Svetlov added the comment: `run` should be awaitable method, see #38430 -- ___ Python tracker ___ ___ Python-bugs-list

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: So, here's a prototype implementation of asyncio.ThreadPool that would function exactly as Yury described, but I'm not convinced about the design. Usually, it seems preferred to separate the context manager from the rest of the class (as was done with

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > end up adding two high-level functions Clarification: asyncio.run_in_executor() would be a function, but asyncio.ThreadPool would be a context manager class. -- ___ Python tracker

[issue32309] Implement asyncio.run_in_executor shortcut

Kyle Stanley added the comment: > I don't like the low-level API of run_in_executor. "executor" being the > first argument, the inability to pass **kwargs, etc. > I mean it's great that we can use 'concurrent.futures' in asyncio, but having > native asyncio pools implementation would be

[issue32309] Implement asyncio.run_in_executor shortcut

Andrew Svetlov added the comment: In Python 3.7 loop.run_in_executor() is the only user-faced method that requires a loop. asyncio.ThreadPool() sounds great. Maybe thread group can provide better api. But for Python 3.8 adding `run_in_executor` top-level function looks the only easy and

[issue32309] Implement asyncio.run_in_executor shortcut

Andrew Svetlov added the comment: https://bugs.python.org/issue32311 opened for create_task() -- ___ Python tracker ___

[issue32309] Implement asyncio.run_in_executor shortcut

Andrew Svetlov added the comment: Removed create_task() from title -- title: Implement asyncio.create_task() and asyncio.run_in_executor shortcuts -> Implement asyncio.run_in_executor shortcut ___ Python tracker