[issue19809] Python get stuck in second Popen call
New submission from Owen Lin:
If we call two subprocess.Popen simultaneously, the second one is blocked until
the first one is finished.
The attached file is a code snippet to reproduce this bug. I can reproduce the
bug in version 2.7.3 and 2.7.6 very easily (in few seconds with the code). But
it works fine on python3.
Here is the backtrace of python
==
#0 0x7f0eba954d2d in read () at ../sysdeps/unix/syscall-template.S:82#1
0x005d8d10 in posix_read (self=0x0, args=(5, 1048576)) at
../Modules/posixmodule.c:6628#2 0x00486896 in PyCFunction_Call
(func=built-in function read, arg=(5, 1048576), kw=0x0)at
../Objects/methodobject.c:81#3 0x005278e4 in ext_do_call
(func=built-in function read, pp_stack=0x7fff1fc0ac80, flags=1, na=0, nk=0)
at ../Python/ceval.c:4331
#4 0x005215cd in PyEval_EvalFrameEx (
f=Frame 0x298f800, for file /usr/lib/python2.7/subprocess.py, line 478, in
_eintr_retry_call (func=built-in funct
ion read, args=(5, 1048576)), throwflag=0) at ../Python/ceval.c:2705
#5 0x00523c2e in PyEval_EvalCodeEx (co=0x294b880,
globals={'STDOUT': -2, '_has_poll': True, 'gc': module at remote 0x29672d0,
'check_call': function at remote 0x29c4450, 'mswindows': False, 'select':
module at remote 0x29676e0, 'list2cmdline': function at remote 0x29c45a0,
'__all__': ['Popen', 'PIPE', 'STDOUT', 'call', 'check_call', 'check_output',
'CalledProcessError'], 'errno': module at remote 0x272d4d8, '_demo_posix':
function at remote 0x29c4648, '__package__': None, 'PIPE': -1, '_cleanup':
func
===
The fd 5 is actually a pipe. But I cannot find the other end of the pipe. A
workaround is using lock around all the Popen()s.
--
files: test.py
messages: 204568
nosy: owenlin
priority: normal
severity: normal
status: open
title: Python get stuck in second Popen call
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file32868/test.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19809
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19809] Python get stuck in second Popen call
STINNER Victor added the comment: The creation of slave_popen is not protected by a lock, and so dummy_thread() may spawn a new process (master) at the same time than the main process (slave). If it occurs at the same time, the master may inherit a pipe of the slave process used internally by the subprocess module to send the exception to the parent process if a Python exception occurs in the child process. The inheritance of pipes has been partially fixed in Python 3.2 with the new function _posixsubprocess.cloexec_pipe() which uses pipe2() function to atomically create a pipe with the O_CLOEXEC flag set. The problem has been fixed complelty in Python 3.4 with the PEP 446: all files are now created non inheritable by default, and atomic functions to set the non inheritable flag are used when available. You posted a Python 2.7 script, so I suppose that you are stuck at Python 2. In this case, you can workaround the issue by using a lock around the creation of any subprocess. To fix your example, just surround slave_popen creation with with lock: ... or lock.acquire()/lock.release(), as you did for master_popen. I propose to convert this issue to a documentation issue: we should add a warning explaining that spawning processes in more than one thread can lead to such race condition and hang one or more threads. And suggest to use a lock to workaround such issue. See also the atfork proposition which includes such lock: http://bugs.python.org/issue16500 -- Your script has other issues: - you should pass a file open in write mode for stdout/stderr - you should close open(os.devnull) files, or open them outside the loop - you should join the thread, or use a daemon thread. If you don't join threads, you will quickly read the limit of the number of threads - you can use shell=True to avoid spawning two process to run sleep (or simply use time.sleep :-)) - you need a synchronization between the two threads to ensure that master_popen is created before trying to kill it -- nosy: +haypo, neologix, sbt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19809 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
