Kyle Stanley <aeros...@gmail.com> added the comment:
So, essentially, are you looking for a way to dynamically adjust ProcessPoolExecutor's (PPE) max_workers, rather than just upon initialization? This seems like it would be a very reasonable enhancement to the Executor API. Specifically for PPE, it would involve expanding the call queue, which is where pending work items are moved to just before being executed by the subprocesses (and then moved to the results queue). In order for futures submitted to the executor to be cancel-able, it has to have a fixed upper limit based on the number of max_workers (but not too low so it doesn't have idle workers). Fortunately, we are able to adjust the *maxsize* of the call queue in real-time since it's part of the public API for Queue (rather than having to create a new queue and copy all of the elements over to a new one). Also, in order for this to be of real benefit, we would have to join all of the processes that are no longer being used. This would likely take some time to implement properly, but without doing so, there wouldn't be a whole lot of real benefit from being able to dynamically adjust the max workers; you'd still be using up the resources to keep a bunch of idle processes alive. I also suspect that it would require fairly extensive tests to ensure it's stability, and would be decently involved to implement it in a way that doesn't negatively impact or modify the existing behavior for ProcessPoolExecutor. But with some time and effort, I suspect it would be possible to implement this feature. Assuming Antoine P. and Brian Q. (primary maintainers/experts for concurrent.futures) are on-board with implementing this feature, I would be willing to look into it working on it when I get the chance to. ---------- nosy: +aeros _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue39617> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
