[issue31935] subprocess.run() timeout not working with grandchildren and stdout=PIPE
New submission from Martin Ritter : Hi, I tried to use subprocess.run(..., stdout=subprocess.PIPE, timeout=N) to run some test scripts with a given timeout. This works as expected with simple scripts. However if the script itself creates other children which write to stdout then `subprocess.run()` seems to wait for all of the children to finish. I've attached a minimal example. I looked into subprocess.py and `subprocess.run()` calls `process.communicate()` again without timeout when handling the TimeoutExpired exception which then in turn waits for the pipes to be closed by all children. If communicate() would check if the process is still alive while waiting for output and close the pipes once the process has finished the timeout feature should work as expected and descendants would get a SIGPIPE when writing to stdout/stderr. -- components: Library (Lib) files: test_killsub.py messages: 305487 nosy: Martin Ritter priority: normal severity: normal status: open title: subprocess.run() timeout not working with grandchildren and stdout=PIPE type: behavior versions: Python 3.6 Added file: https://bugs.python.org/file47251/test_killsub.py ___ Python tracker <https://bugs.python.org/issue31935> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27422] Deadlock when mixing threading and multiprocessing
Martin Ritter added the comment: Dear Davin, Thanks for the input, I was perfectly aware that the "solution" I proposed is not realistic. But the feedback that multiprocessing is using threads internally is useful as I can quickly abandon the idea to do something like the check I proposed in our code base without spending time on it. I was aware of the Gil, I just did not anticipate that big a problem when mixing threads and processes with rather simple python code. My bad, sorry for the noise. Cheers, Martin -- ___ Python tracker <http://bugs.python.org/issue27422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27422] Deadlock when mixing threading and multiprocessing
Martin Ritter added the comment: I agree that this is error prone and can not be fixed reliably on the python side. However, python makes it very easy to mix these two, a user might not even notice it if a function he calls uses fork and thus just use a ThreadPoolExecutor() because it's the simplest thing to do. What could be an nice solution in my opinion if the multiprocessing module could check if there are already multiple threads active on process creation and issue a warning if so. This warning could of course be optional but would make this issue more obvious. In my case we have a large C++ code base which still includes a lot of Fortran 77 code with common blocks all over the place (yay science). Everything is interfaced in python so to make sure that I do not have any side effects I run the some of the functions in a fork using multiprocessing.Process(). And in this case I just wanted to run some testing in parallel. I now switched to a ProcessPoolExecutor which works fine but for me. -- ___ Python tracker <http://bugs.python.org/issue27422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23644] g++ module compile fails with ‘_Atomic’ does not name a type
Martin Ritter added the comment: For me compiling 3.5.2 on GCC 5.2.0 fails when I supply --with-cxx-main=g++ with the same "error: ‘_Atomic’ does not name a type" messages. I'm guessing that --with-cxx-main is probably outdated(?) but at least it worked with 3.5.0 ------ nosy: +Martin Ritter ___ Python tracker <http://bugs.python.org/issue23644> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27422] Deadlock when mixing threading and multiprocessing
Martin Ritter added the comment: I attached a gdb backtrace of one of the child processes -- Added file: http://bugs.python.org/file43589/test_threadfork_backtrace.txt ___ Python tracker <http://bugs.python.org/issue27422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27422] Deadlock when mixing threading and multiprocessing
New submission from Martin Ritter: When creating a multiprocessing.Process in a threaded environment I get deadlocks waiting, I guess waiting for the lock to flush the output. I attached a minimal example of the problem which hangs for me starting with 4 threads. -- files: test_threadfork.py messages: 269593 nosy: Martin Ritter priority: normal severity: normal status: open title: Deadlock when mixing threading and multiprocessing type: crash versions: Python 3.5 Added file: http://bugs.python.org/file43588/test_threadfork.py ___ Python tracker <http://bugs.python.org/issue27422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
