[issue18966] Threads within multiprocessing Process terminate early

Antoine Pitrou added the comment: This is now fixed in git master. Thank you for the report! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue18966] Threads within multiprocessing Process terminate early

Antoine Pitrou added the comment: New changeset ee84a608587b930176d37303afae8a4358e15990 by Antoine Pitrou in branch 'master': bpo-18966: non-daemonic threads created by a multiprocessing.Process should be joined on exit (#3111)

[issue18966] Threads within multiprocessing Process terminate early

Changes by Antoine Pitrou : -- stage: needs patch -> patch review versions: +Python 3.7 -Python 3.6 ___ Python tracker ___

[issue18966] Threads within multiprocessing Process terminate early

Changes by Antoine Pitrou : -- pull_requests: +3150 ___ Python tracker ___ ___

[issue18966] Threads within multiprocessing Process terminate early

Tim Peters added the comment: About: "The notion of categorically refusing to let a process end perhaps overreaches in certain situations." threading.py addressed that all along: if the programmer _wants_ the process to exit without waiting for a particular threading.Thread, that's fine,

[issue18966] Threads within multiprocessing Process terminate early

R. David Murray added the comment: As far as muliprocessing's "mentality" goes, it aims to provide the *same* API as Threading, so it is logical that it should preserve threading's behavior with respect to child threads in a process, rather than violating threading's model. Anything else is

[issue18966] Threads within multiprocessing Process terminate early

R. David Murray added the comment: Note, however, that fixing this will be a backward compatibility issue, since there are doubtless programs relying on this behavior, probably mostly unintentionally. -- ___ Python tracker

[issue18966] Threads within multiprocessing Process terminate early

Eryk Sun added the comment: > all atexit handlers, for example, would be called multiple times. Davin is (I think) proposing a multiprocessing atexit facility, which can be used to ensure threading._shutdown is called. But could Python's regular atexit handling be reset in the child, allowing

[issue18966] Threads within multiprocessing Process terminate early

Davin Potts added the comment: Tim: Totally agreed about threading.Thread not being a POSIX thread. It was not my intent to suggest that they were equivalent -- apologies for the confusion. Instead I was attempting to describe a mentality of processes and their common behavior across

[issue18966] Threads within multiprocessing Process terminate early

Charles-François Natali added the comment: One reason for not calling sys.exit() is because on Linux, the default implementation uses fork(), hence the address space in the chilren is a clone of the parent: so all atexit handlers, for example, would be called multiple times. There's also the

[issue18966] Threads within multiprocessing Process terminate early

Changes by Antoine Pitrou : -- stage: -> needs patch type: behavior -> enhancement versions: -Python 2.7, Python 3.5 ___ Python tracker

[issue18966] Threads within multiprocessing Process terminate early

Antoine Pitrou added the comment: I agree with Tim. Regardless of what OS threads do, Python tries to enforce predictable semantics of its own. There's no reason (apart from historical baggage) to not join Python threads (and only Python threads, of course, not other OS threads) at the

[issue18966] Threads within multiprocessing Process terminate early

Tim Peters added the comment: About ""No parents, no children", that's fine so far as it goes. But Python isn't C, a threading.Thread is not a POSIX thread, and threading.py _does_ have a concept of "the main thread". There's no conceptual problem _in Python_ with saying "the main thread"

[issue18966] Threads within multiprocessing Process terminate early

Eryk Sun added the comment: > Per Eryk's point about the difference in multiprocessing's behavior > when using spawn vs. fork, the explanation for why it's done that > way is also described in the DeveloperWorks article I mentioned above. Please spell this out for me. Why can't the "fork" and

[issue18966] Threads within multiprocessing Process terminate early

Tim Peters added the comment: Devin, a primary point of `threading.py` is to provide a sane alternative to the cross-platform thread mess. None of these reports are about making it easier for threads to go away "by magic" when the process ends. It's the contrary: they're asking for

[issue18966] Threads within multiprocessing Process terminate early

Davin Potts added the comment: It is a general rule that when a process terminates, it likewise terminates all its threads (unless a thread has been explicitly detached). How it goes about doing so is complicated. Remember that POSIX threads have no concept of parent/child among themselves

[issue18966] Threads within multiprocessing Process terminate early

Changes by Ned Deily : -- nosy: +davin ___ Python tracker ___ ___ Python-bugs-list mailing

[issue18966] Threads within multiprocessing Process terminate early

Eryk Sun added the comment: In 3.4+ it works correctly with the "spawn" start method. This uses multiprocessing.spawn.spawn_main, which exits the child via sys.exit(exitcode). "fork" and "forkserver" exit the child via os._exit(code), respectively in multiprocessing.popen_fork.Popen._launch

[issue18966] Threads within multiprocessing Process terminate early

Tim Peters added the comment: This came up again today as bug 27508. In the absence of "fixing it", we should add docs to multiprocessing explaining the high-level consequences of skipping "normal" exit processing (BTW, I'm unclear on why it's skipped). I've certainly mixed threads with

[issue18966] Threads within multiprocessing Process terminate early

Antoine Pitrou added the comment: That's because multiprocessing exits child processes with os._exit(), not sys.exit(). The fix would be trivial (call threading._shutdown() before os._exit()), but I don't know if that's something we want to do. After all there are many things in the Python

[issue18966] Threads within multiprocessing Process terminate early

New submission from Piet van Oostrum: When a process started as a multiprocessing Process spawns a thread, it doesn't wait until the thread terminates. It terminates the thread early when the main thread of the process terminates, as if the thread would be daemonic (it isn't). It may sound a