[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Gregory P. Smith added the comment: fyi - this appears to have caused https://bugs.python.org/issue46464 -- nosy: +gregory.p.smith ___ Python tracker ___ _

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

miss-islington added the comment: New changeset 1ac6e379297cc1cf8acf6c1b011fccc7b3da2cbe by Kyle Stanley in branch 'master': bpo-39207: Spawn workers on demand in ProcessPoolExecutor (GH-19453) https://github.com/python/cpython/commit/1ac6e379297cc1cf8acf6c1b011fccc7b3da2cbe -- nosy

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Change by Antoine Pitrou : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ __

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Change by Kyle Stanley : -- keywords: +patch pull_requests: +18807 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/19453 ___ Python tracker

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Kyle Stanley added the comment: So, I think a potential approach to this issue with ProcessPoolExecutor would be making it a bit more similar to ThreadPoolExecutor: instead of spawning *max_workers* processes on startup (see `_adjust_process_count()`), spawn each new process in submit() (see

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Kyle Stanley added the comment: > It would certainly be better to start the worker processes on demand. It > probably also requires careful thought about how to detect that more workers > are required. Alright. In that case, I'll do some experimentation when I get the chance and see if I c

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Antoine Pitrou added the comment: It would certainly be better to start the worker processes on demand. It probably also requires careful thought about how to detect that more workers are required. -- ___ Python tracker

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Kyle Stanley added the comment: > I think this should be fixed like ThreadPoolExecutor. Are there are any downsides or complications with changing this behavior for ProcessPoolExecutor to consider, such as what I mentioned above? From my understanding, there would be a performance penalty as

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Inada Naoki added the comment: Uh, my understanding "But ProcessPoolExecutor starts worker processes on demand from old." was wrong. I think this should be fixed like ThreadPoolExecutor. -- ___ Python tracker

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Kyle Stanley added the comment: > What "ignores the max_workers argument" means? >From my understanding, their argument was that the parameter name >"max_workers" and documentation implies that it will spawn processes as needed >up to *max_workers* based on the number of jobs scheduled. >

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Change by Terry J. Reedy : -- nosy: +bquinlan, pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

Inada Naoki added the comment: > ProcessPoolExecutor does not properly spin down and spin up new processes. It is because Process "Pool" is for reusing processes. If you don't want to reuse process, you can use the Process. https://docs.python.org/3/library/multiprocessing.html#the-process-cla

[issue39207] concurrent.futures.ProcessPoolExecutor does not properly reap jobs and spawns too many workers

New submission from Yusef Shaban : This came up from a supporting library but the actual issue is within concurrent.futures.ProcessPool. Discussion can be found at https://github.com/agronholm/apscheduler/issues/414 ProcessPoolExecutor does not properly spin down and spin up new processes. I