[issue27422] Deadlock when mixing threading and multiprocessing

R. David Murray added the comment: Heh, yeah. What I was really trying to do by that comment was clarify for any *other* readers that stumble on this issue later it is just the python code that *has* to be constrained by the GIL. I have no idea how much of the scipy stack drops the gil at

[issue27422] Deadlock when mixing threading and multiprocessing

Changes by Davin Potts : -- stage: -> resolved status: open -> closed ___ Python tracker ___

[issue27422] Deadlock when mixing threading and multiprocessing

Davin Potts added the comment: @r.david.murray: Oh man, I was not going to go as far as advocate dropping the GIL. :) At least not in situations like this where the exploitable parallelism is meant to be at the Python level and not inside the Fortran code (or that was my understanding of

[issue27422] Deadlock when mixing threading and multiprocessing

R. David Murray added the comment: To clarify the GIL issue (for davin, I guess? :): if the library you are using to interface with the FORTRAN code drops the GIL before calling the FORTRAN, then you *can* take advantage of multiple cores. It is only the python code (and some of the code

[issue27422] Deadlock when mixing threading and multiprocessing

Martin Ritter added the comment: Dear Davin, Thanks for the input, I was perfectly aware that the "solution" I proposed is not realistic. But the feedback that multiprocessing is using threads internally is useful as I can quickly abandon the idea to do something like the check I proposed

[issue27422] Deadlock when mixing threading and multiprocessing

Davin Potts added the comment: While I believe I understand the motivation behind the suggestion to detect when the code is doing something potentially dangerous, I'll point out a few things: * any time you ask for a layer of convenience, you must choose something to sacrifice to get it

[issue27422] Deadlock when mixing threading and multiprocessing

Martin Ritter added the comment: I agree that this is error prone and can not be fixed reliably on the python side. However, python makes it very easy to mix these two, a user might not even notice it if a function he calls uses fork and thus just use a ThreadPoolExecutor() because it's the

[issue27422] Deadlock when mixing threading and multiprocessing

Raymond Hettinger added the comment: FWIW, this isn't even a Python specific behavior. It is just how threads, locks, and processes work (or in this case don't work). The code is doing what it is told to do which happens to not be what you want (i.e. a user bug rather than a Python bug). I

[issue27422] Deadlock when mixing threading and multiprocessing

Davin Potts added the comment: It would be nice to find an appropriate place to document the solid general guidance Raymond provided; though merely mentioning it somewhere in the docs will not translate into it being noticed. Not sure where to put it just yet... Martin: Is there a specific

[issue27422] Deadlock when mixing threading and multiprocessing

Raymond Hettinger added the comment: It is in-fact problem prone (and not just in Python). The rule is "thread after you fork, not before". Otherwise, the locks used by the thread executor will get duplicated across processes. If one of those processes dies while it has the lock, all of

[issue27422] Deadlock when mixing threading and multiprocessing

R. David Murray added the comment: Mixing multiprocessing and threading is problem prone in general. Hopefully one of the multiprocessing experts can say if this is a known problem or not... -- nosy: +devin, r.david.murray, sbt ___ Python tracker

[issue27422] Deadlock when mixing threading and multiprocessing

Martin Ritter added the comment: I attached a gdb backtrace of one of the child processes -- Added file: http://bugs.python.org/file43589/test_threadfork_backtrace.txt ___ Python tracker

[issue27422] Deadlock when mixing threading and multiprocessing

New submission from Martin Ritter: When creating a multiprocessing.Process in a threaded environment I get deadlocks waiting, I guess waiting for the lock to flush the output. I attached a minimal example of the problem which hangs for me starting with 4 threads. -- files: