[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Michał Górny added the comment: After updating PyPy3 to use Python 3.9's stdlib, we hit very bad hangs because of this — literally compiling a single file with "parallel" compileall could hang. In the end, we had to revert the change in how Python 3.9 starts workers because otherwise

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Change by wim glenn : -- nosy: +wim.glenn ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Antoine Pitrou added the comment: Agreed, but again, changing will break some applications. We could switch to forkserver, but we should have a transition period where a FutureWarning will be displayed if people didn't explicitly set a start method. --

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Itamar Turner-Trauring added the comment: Given people's general experience, I would not say that "fork" works on Linux either. More like "99% of the time it works, 1% it randomly breaks in mysterious way". -- ___ Python tracker

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Antoine Pitrou added the comment: The macOS change was required before "fork" simply ceased to work. Windows has always used "spawn", because no other method can be implemented on Windows. -- ___ Python tracker

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Itamar Turner-Trauring added the comment: This change was made on macOS at some point, so why not Linux? "spawn" is already the default on macOS and Windows. -- ___ Python tracker

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Antoine Pitrou added the comment: The problem with changing the default is that this will break any application that depends on passing non-picklable data to the child process (in addition to the potentially unexpected performance impact). The docs already contain a significant elaboration

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Andrew Duncan added the comment: I just ran into and fixed (thanks to itamarst's blog post) a problem likely related to this. Multiprocessing workers performing work and sending a logging message back with success/fail info. I had a few intermittent deadlocks that became a recurring

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Itamar Turner-Trauring added the comment: Another person with the same issue: https://twitter.com/volcan01010/status/1324764531139248128 -- nosy: +itamarst2 ___ Python tracker

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Change by Julian Berman : -- nosy: +Julian ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Change by Ned Deily : -- nosy: +davin, pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Itamar Turner-Trauring added the comment: Just got an email from someone for whom switching to "spawn" fixed a problem. Earlier this week someone tweeted about this fixing things. This keeps hitting people in the real world. -- ___ Python tracker

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

Itamar Turner-Trauring added the comment: Looks like as of 3.8 this only impacts Linux/non-macOS-POSIX, so I'll amend the above to say this will also make it consistent with macOS. -- ___ Python tracker

[issue40379] multiprocessing's default start method of fork()-without-exec() is broken

New submission from Itamar Turner-Trauring : By default, multiprocessing uses fork() without exec() on POSIX. For a variety of reasons this can lead to inconsistent state in subprocesses: module-level globals are copied, which can mess up logging, threads don't survive fork(), etc.. The end