[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Change by Myles Steinhauser : -- nosy: +myles.steinhauser ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Change by Ruairidh MacLeod : -- nosy: +rkm ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Change by Jon Clucas : -- nosy: +shnizzedy ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Marko added the comment: Somewhat related issue43806 with asyncio.StreamReader -- ___ Python tracker ___ ___ Python-bugs-list

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Marko added the comment: I've created issue43805. I think it would be better to have universal solution. And not specific ones, like in issue9205. Haven't checked the PRs, though. -- nosy: +kormang ___ Python tracker

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Change by Davin Potts : -- pull_requests: +15722 pull_request: https://github.com/python/cpython/pull/16103 ___ Python tracker ___

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

STINNER Victor added the comment: I just marked bpo-38084 as duplicate of this issue. I manually merged the nosy lists. -- nosy: +vstinner ___ Python tracker ___

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Chris Markiewicz added the comment: Just a bump to note that the PR (10441) is ready for another round of review. -- nosy: +cjmarkie ___ Python tracker ___

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Oscar Esteban added the comment: I tried to reuse as much as I could from the patch, but it didn't solve the issue at first. I have changed the responsibility of identifying and prescribing a solution when a worker got killed. In the proposed patch, the thread handling results (i.e. tasks

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Change by Oscar Esteban : -- pull_requests: +9713 stage: needs patch -> patch review ___ Python tracker ___ ___ Python-bugs-list

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Antoine Pitrou added the comment: You should start from master. Bugfixes can backported afterwards if appropriate. Thanks! -- ___ Python tracker ___

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Oscar Esteban added the comment: Hi Antoine, I may take a stab at it. Before I start, should I branch from master or from 3.7.1 (as 3.7 is still accepting bugfixes). Best, Oscar -- ___ Python tracker

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Antoine Pitrou added the comment: Oscar, the patch posted here needs updating for the latest git master. If you want to avoid this issue, you can also use concurrent.futures where the issue is fixed. -- stage: -> needs patch versions: +Python 3.8 -Python 3.5

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Oscar Esteban added the comment: We use multiprocessing to parallelize many tasks that run either python code or call subprocess.run that are memory hungry. At times the OOM Killer kicks in. When one of the workers is killed, the queue never returns an error for the

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Francis Bolduc added the comment: This problem also happens simply by calling sys.exit from one of the child processes. The following script exhibits the problem: import multiprocessing import sys def test(value): if value: sys.exit(123) if __name__ == '__main__': pool =

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Changes by Davin Potts : -- nosy: +davin ___ Python tracker ___ ___

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Changes by Brian Boonstra : -- nosy: +brianboonstra ___ Python tracker ___ ___

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

Changes by Chris Rebert pyb...@rebertia.com: -- nosy: +cvrebert ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22393 ___ ___ Python-bugs-list

[issue22393] multiprocessing.Pool shouldn't hang forever if a worker process dies unexpectedly

New submission from Dan O'Reilly: This is essentially a dupe of issue9205, but it was suggested I open a new issue, since that one ended up being used to fix this same problem in concurrent.futures, and was subsequently closed. Right now, should a worker process in a Pool unexpectedly get