Re: Using subprocess module to launch a shell shell script that itself forks a process
On Tue, Oct 07, 2008 at 05:43:41PM -0700, Samuel A. Falvo II wrote: p = subprocess.Popen( command, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True ) outputChannel = p.stdout output = outputChannel.read() You call read with no arguments. This is highly problematic in the context of interprocess communication, unless you can be 100% positive that none of the children will write anywhere besides STDOUT, and won't try to read from STDIN in the meanwhile. Python's read() with no args reads until the end of file, which in IPC contexts is bad... Normally the child process won't close stdout until it exits. So, if it did any other output in between, say, to STDERR, the child will block waiting for the parent to read STDERR, meanwhile the parent is blocked waiting for input from the child's STDOUT, which results in a deadlock. Both processes sleep forever. The exact same thing can happen if either the shell script or a process started by the shell script tries to read from STDIN. Since Java is launched by the shell script, it inherits the shell script's STDIN, STDOUT, and STDERR file descriptors (i.e. the two processes share the same STDIO). Thus if the java process writes to STDERR, that also could be causing your deadlock. On Wed, Oct 08, 2008 at 11:24:39AM -0700, Samuel A. Falvo II wrote: On Oct 7, 6:23 pm, Gabriel Genellina [EMAIL PROTECTED] wrote: Is your shell script doing something else, apart from invoking the java process? Obviously, yes. It's far from obvious. I can't count the number of scripts I've seen whose sole purpose was to launch a Java program (with the right environment)... You would do well to avoid being dismissive and listen to those who respond with help, when you are the one who obviously doesn't understand the behavior you're getting, and you're the one asking for help. The script is some 150 lines long. But the hang-up occurs because of the forked Java process, not the other lines. I'm betting it's because the Java program is writing warnings to STDERR (more data than will fit in the STDIO buffer), which you're not reading... If not, you could just invoke java directly from Python. Also, you set stdin=PIPE - is your java process expecting some input? you're not writing anything to stdin. It does not expect input from stdin. However, this does not affect any OTHER scripts or commands I run. Irrelevant... Unless any OTHER scripts encompases all possible combinations of process interactions, and you can demontstrate that it does so, this proves nothing. Let's remember to look at the objective facts: for shell scripts that launch child processes of their own, Python hangs. For all other types of commands, it works 100% as expected. These are not facts which are in evidence. We don't know what your script is doing, and it's clear that you yourself are not able to explain the behavior you are seeing, therefore there is no reason for us to conclude that the above statements are true and correct. Most likely, they are not. Anyway, it's better to use the communicate method instead (it uses select to read from both stdout and stderr): That doesn't help me. Please explain why it doesn't. The most likely cause of the behavior you are seeing is the deadlock I described, above. Using select() (i.e. using communicate()) should generally fix about half the cases... The rest would be fixed by redirecting STDIN of the child (or at least the Java process) from /dev/null, most likely. Then of course, there could be other explanations. But this being overwhelmingly the most likely one, if you don't try those solutions, there's no point in trying to help you further... -- Derek D. Martin http://www.pizzashack.org/ GPG Key ID: 0x81CFE75D pgp8zdVYVQEhV.pgp Description: PGP signature -- http://mail.python.org/mailman/listinfo/python-list
Re: Using subprocess module to launch a shell shell script that itself forks a process
On Oct 7, 6:23 pm, Gabriel Genellina [EMAIL PROTECTED] wrote: Is your shell script doing something else, apart from invoking the java process? Obviously, yes. The script is some 150 lines long. But the hang-up occurs because of the forked Java process, not the other lines. If not, you could just invoke java directly from Python. Also, you set stdin=PIPE - is your java process expecting some input? you're not writing anything to stdin. It does not expect input from stdin. However, this does not affect any OTHER scripts or commands I run. Let's remember to look at the objective facts: for shell scripts that launch child processes of their own, Python hangs. For all other types of commands, it works 100% as expected. Anyway, it's better to use the communicate method instead (it uses select to read from both stdout and stderr): That doesn't help me. See http://docs.python.org/library/subprocess.html#subprocess.Popen.commu... I have. -- http://mail.python.org/mailman/listinfo/python-list
Re: Using subprocess module to launch a shell shell script that itself forks a process
On Oct 8, 11:31 am, Samuel A. Falvo II [EMAIL PROTECTED] wrote: I removed the stdin=PIPE argument, and this works. Many thanks for bringing this to my attention. OK, I am confused. After observing a bug where the code works every other time, like clockwork, I've used strace to figure out what is going on. In the times when it works, it's only the shell script complaining that it's already running. So, I'm right back to square one. Even _without_ setting stdin=PIPE, it fails to unblock. I'm left utterly bewildered. The only thing I can think of is that the JVM's stdout is tied to the shell script's stdout, because that's the only way it can remain open upon the termination of the child process. I suppose, at this point, the next place to look is in shell script syntax to find out how to detach the JVM from the script's process group. -- http://mail.python.org/mailman/listinfo/python-list
Re: Using subprocess module to launch a shell shell script that itself forks a process
On Oct 8, 11:24 am, Samuel A. Falvo II [EMAIL PROTECTED] wrote: On Oct 7, 6:23 pm, Gabriel Genellina [EMAIL PROTECTED] wrote: Is your shell script doing something else, apart from invoking the java process? Obviously, yes. The script is some 150 lines long. But the hang-up occurs because of the forked Java process, not the other lines. If not, you could just invoke java directly from Python. Also, you set stdin=PIPE - is your java process expecting some input? you're not writing anything to stdin. It does not expect input from stdin. However, this does not affect any OTHER scripts or commands I run. Let's remember to look at the objective facts: for shell scripts that launch child processes of their own, Python hangs. For all other types of commands, it works 100% as expected. Anyway, it's better to use the communicate method instead (it uses select to read from both stdout and stderr): That doesn't help me. See http://docs.python.org/library/subprocess.html#subprocess.Popen.commu... I have. You should be nicer to Gabriel. He is a guru. -- http://mail.python.org/mailman/listinfo/python-list
Re: Using subprocess module to launch a shell shell script that itself forks a process
On Oct 8, 11:24 am, Samuel A. Falvo II [EMAIL PROTECTED] wrote: It does not expect input from stdin. However, this does not affect any OTHER scripts or commands I run. OK, so, I'm very confused as to why this would matter. I removed the stdin=PIPE argument, and this works. Many thanks for bringing this to my attention. But, my question now is, WHY is this an issue? If the launched process doesn't read from its stdin, why would it block? Hence my question here: Let's remember to look at the objective facts: for shell scripts that launch child processes of their own, Python hangs. For all other types of commands, it works 100% as expected. Meaning, if I launched the Java process directly, it works fine. If I launch it from the shell script WITHOUT background execution, it works fine. But when I launch it WITH background execution (e.g., with the suffix), then it blocks. Any ideas, so I can write this into my log for future reference? Thanks. -- http://mail.python.org/mailman/listinfo/python-list
Re: Using subprocess module to launch a shell shell script that itself forks a process
En Wed, 08 Oct 2008 15:24:39 -0300, Samuel A. Falvo II
[EMAIL PROTECTED] escribió:
On Oct 7, 6:23 pm, Gabriel Genellina [EMAIL PROTECTED] wrote:
you set stdin=PIPE - is your java process expecting some input? you're
not writing anything to stdin.
It does not expect input from stdin. However, this does not affect
any OTHER scripts or commands I run.
But it *does* affect how subprocess handles the child process internally.
Let's remember to look at the objective facts: for shell scripts that
launch child processes of their own, Python hangs. For all other
types of commands, it works 100% as expected.
Don't conclude too fast... I've tested a variant (using a Perl script as a
replacement instead of a Java program), and it worked fine:
log
[EMAIL PROTECTED]:~$ cat foo.sh
#!/bin/sh
perl foo.pl
echo End of foo.sh
[EMAIL PROTECTED]:~$ cat foo.pl
#!/usr/bin/perl
for ($i=5; $i0; $i--) {
print $i seconds remaining...\n;
sleep(1);
}
print End of foo.pl\n;
[EMAIL PROTECTED]:~$ cat foo.py
#!/usr/bin/python2.5
import subprocess
command = ./foo.sh
p = subprocess.Popen(command,
shell=True,
stdin=None,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
close_fds=True
)
print py: before p.communicate
output = p.communicate()[0]
print py: after p.communicate
print py: returncode=,p.returncode
print py: read,len(output),bytes
print py: output=, repr(output)
print py: End of foo.py
[EMAIL PROTECTED]:~$ ./foo.py
py: before p.communicate
py: after p.communicate
py: returncode= 0
py: read 143 bytes
py: output= 'End of foo.sh\n5 seconds remaining...\n4 seconds
remaining...\n3 seconds remaining...\n2 seconds remaining...\n1 seconds
remaining...\nEnd of foo.pl\n'
py: End of foo.py
[EMAIL PROTECTED]:~$ uname -a
Linux debian 2.6.18-4-486 #1 Mon Mar 26 16:39:10 UTC 2007 i686 GNU/Linux
[EMAIL PROTECTED]:~$
/log
This was tested both with python 2.4.4 and 2.5.2
I don't think it's a Python problem, but something related to java and how
it handles stdin/stdout. Try with another program.
But, my question now is, WHY is this an issue? If the launched
process doesn't read from its stdin, why would it block?
Several reasons - the child process might send enough text to stderr to
fill its buffer, and if the parent process doesn't read from the other end
of the pipe, the child blocks. That's why I suggested to use communicate
instead of stdout.read() - it takes care of such cases.
The only thing I can think of is that the JVM's stdout is tied to the
shell script's stdout, because that's the only way it can remain open
upon the termination of the child process. I suppose, at this point,
the next place to look is in shell script syntax to find out how to
detach the JVM from the script's process group.
Can't you redirect to a file instead?
java foo.jar /tmp/foo.log 21
--
Gabriel Genellina
--
http://mail.python.org/mailman/listinfo/python-list
Using subprocess module to launch a shell shell script that itself forks a process
I have a shell script script.sh that launches a Java process in the background using the -operator, like so: #!/bin/bash java ... arguments here ... In my Python code, I want to invoke this shell script using the Subprocess module. Here is my code: def resultFromRunning_(command): Invokes a shell command, and returns the stdout response. Args: command: A string containing the complete shell command to run. Results: A string containing the output of the command executed. Raises: ValueError if a non-zero return code is returned from the shell. OSError if command isn't found, inappropriate permissions, etc. L = log4py.Logger().get_instance() L.info(Executing: + command) p = subprocess.Popen( command, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True ) outputChannel = p.stdout output = outputChannel.read() result = p.wait() if result: raise(ShellError(command, result, output)) L.info(Result = + str(output)) return output When running the aforementioned code, it kicks off the shell script, and, the shell script kicks off the Java process. However, the Python code never returns from outputChannel.read() until I explicitly kill the Java process myself via the kill shell command. I've researched this issue on Google and various other websites, and maybe I'm missing the obvious, but I cannot seem to find any documentation relevant to this problem. Lots of references to bugs filed in the past, that appear to be fixed, or to websites talking about how the Popen module has a 64K limit on its data queue size, but nothing relevent to my situation. Can anyone inform me or point me to the appropriate documentation on how to properly invoke a shell command such that any spawned children processes don't cause Python to hang on me? I assume it has something to do with process groups, but I'm largely ignorant of how to control those. Thanks in advance. -- http://mail.python.org/mailman/listinfo/python-list
Re: Using subprocess module to launch a shell shell script that itself forks a process
En Tue, 07 Oct 2008 21:43:41 -0300, Samuel A. Falvo II [EMAIL PROTECTED] escribió: I have a shell script script.sh that launches a Java process in the background using the -operator, like so: #!/bin/bash java ... arguments here ... In my Python code, I want to invoke this shell script using the Subprocess module. Here is my code: def resultFromRunning_(command): Invokes a shell command, and returns the stdout response. Args: command: A string containing the complete shell command to run. Results: A string containing the output of the command executed. Raises: ValueError if a non-zero return code is returned from the shell. OSError if command isn't found, inappropriate permissions, etc. L = log4py.Logger().get_instance() L.info(Executing: + command) p = subprocess.Popen( command, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True ) outputChannel = p.stdout output = outputChannel.read() result = p.wait() if result: raise(ShellError(command, result, output)) L.info(Result = + str(output)) return output When running the aforementioned code, it kicks off the shell script, and, the shell script kicks off the Java process. However, the Python code never returns from outputChannel.read() until I explicitly kill the Java process myself via the kill shell command. Is your shell script doing something else, apart from invoking the java process? If not, you could just invoke java directly from Python. Also, you set stdin=PIPE - is your java process expecting some input? you're not writing anything to stdin. Anyway, it's better to use the communicate method instead (it uses select to read from both stdout and stderr): p = subprocess.Popen(...) output = p.communicate()[0] result = p.returncode if result: ... See http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
