[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
iunknwn <sz2...@gmail.com> added the comment: I feel like there are two reasonable options here: 1) We can implement a thread pool with basic resource tracking. This means idle threads get recycled, and threads that have been sitting idle for a while are terminated as demand drops, so resources can be reclaimed. This wouldn't require very much work - we would just need to modify some of the queue operations to have timeouts, and track the number of idle threads (most of that's already in my first PR). We could easily add options like min_threads and idle_thread_timeout as options in kwargs to the init routine. 2) We can skip all tracking, and spin a fixed number of threads at initialization. This removes the complexity of locks and counts, and means the thread pool executor will work identically to the process pool executor (which also eagerly spawns resources). If we want this, this is ready to go in the second PR. I personally like option 1 because it feels closer to other languages I've worked in, but I'd like a bit more guidance from the reviewers before proceeding. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
iunknwn <sz2...@gmail.com> added the comment: Done - as recommend, I've opened a new PR that changes the behavior to spawn all worker threads when the executor is created. This eliminates all the thread logic from the submit function. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
Change by iunknwn <sz2...@gmail.com>: -- pull_requests: +6224 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
iunknwn <sz2...@gmail.com> added the comment: Alright - I'll put together another patch that removes the logic, and spins up all threads during initialization. Do you want me to create a completely new PR, or just update my existing one? -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
iunknwn <sz2...@gmail.com> added the comment: The existing behavior seems strange (and isn't well documented). The code had a TODO comment from bquinlan to implement idle thread recycling, so that was why I made the change. That said, if threads are cheap, why not just create all the work threads on initialization, and then remove all the logic entirely? Also, regarding the executor and thread-safety, there's an example in the current docs showing a job being added to the executor from a worker thread (it's part of the example on deadlocks, but it focuses on max worker count, not on the executor's thread-safety). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
iunknwn <sz2...@gmail.com> added the comment: I've submitted a PR that should resolve this - it uses a simple atomic counter to ensure new threads are not created if existing threads are idle. One concern I do have - while writing the patch, I noticed the existing submit method (specifically the adjust_thread_count function) isn't thread safe. I've added more details in the PR. -- components: +Library (Lib) ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24882] ThreadPoolExecutor doesn't reuse threads until #threads == max_workers
Change by iunknwn <sz2...@gmail.com>: -- nosy: +iunknwn ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue24882> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
