In article <b166181a-2aba-4c3a-948d-674755459...@c10g2000yqh.googlegroups.com> Roger Davis <r...@hawaii.edu> wrote: >My understanding is that this functionality is best coded via >subprocess.Popen().
"Best" is always a big question mark. :-) >I need to read output from these spawned children >via a pipe from their stdout, hence something like > > p= subprocess.Popen(args, stdout=subprocess.PIPE) > >This means that somewhere a pipe file descriptor is opened on the >parent side to read from the child's stdout. When, if ever, is that >descriptor closed? (I am going to tell this tale in a slightly different order than your question asked, as I think it works out better that way.) subprocess.Popen() creates the instance variable and any pipes needed, forks (on a Unix system) the target process, but has not yet done any I/O with it (except to read a success/fail indicator for whether the exec worked and/or any exception that occurred before then, e.g., during the preexec_fn). It then makes the stdin, stdout, and/or stderr attributes (p.stdout, for the example above) using os.fdopen(). Streams not requested in the call are set to None (so p.stderr, for instance, will be None in this case). At this point, then, the underlying open pipe is still around. But your next step is (normally) to use p.communicate(); this is where most of the magic happens. The Unix implementation loops, using select() to read and write from/to whichever pipe(s) are open to the child process, until *all* data are sent and received. As each data stream is finished, it is closed (in this case, via self.stdout.close()). Lastly, p.communicate() invokes p.wait() (via self.wait()), to wait for the child process to exit. By the time p.communicate() returns, the pipe is closed and the command has finished. The entire output text, however large it is, is returned as the first element of the return-value 2-tuple (remember that p.communicate() returns both the stdout and the stderr -- stderr will be the empty string in this case, as stderr was not redirected in the subprocess.Popen() call). >Per-process FDs are limited and I am looping >infinitely so I need to be very careful about not running out of them. >Are there any other FDs related to this operation that also need to be >closed? Only if you (or code you call) have opened them and not set FD_CLOEXEC. In this case, you can set close_fds = True in your call to subprocess.Popen(). That will make the child of fork() loop over higher-number fd's, calling os.close() on each one. >Testing with the interpreter (2.6, MacOSX) it appears that p.stdout is >being closed somehow by someone other than me: > >import subprocess >args= ["echo", "This is a mystery!"] >i= 0 >while True: > p= subprocess.Popen(args, stdout=subprocess.PIPE) > for line in p.stdout: > print "[%5d] %s" % (i, line.strip()) > i+= 1 > >The above code closes nothing but appears to run indefinitely without >running the parent out of FDs. WTF is going on here? The above reads from p.stdout -- the os.fdopen() result on the underlying pipe -- directly. In the general case (multiple input and output pipes), this is not safe as you can deadlock with constipated pipes (hence the existence of p.communicate()). In this specific case, there is just one pipe so the deadlock issue goes away. Instead, the file descriptor remains open while the inner loop runs (i.e., while "line in p.stdout" is able to fetch lines via the file's iterator). When the loop stops the pipe is still open in the parent, but the child has finished and is now exiting (or has exited or will exit soon). You then reach the "i+=1" line and resume the loop, calling subprocess.Popen() anew. Now we get to the even deeper magic. :-) What happens to the *old* value in p? Answer: because p is reassigned, the (C implementation, interpreted Python bytecode runtime) reference count drops [%]. Since p was the only live reference, the count drops from 1 to 0. This makes the old instance variable go away, invoking old_p.__del__() as it were. The deletion handler cleans up a few things itself, including a a call to os.waitpid() if needed, and then simply lets the reference to old_p.stdout go away. That in turn decrements old_p.stdout's reference count. Since that, too, reaches zero, its __del__ is run ... and *that* closes the underlying file descriptor. [% This is all simplified -- the Python documentation mentions that reference counting for local variables is somewhat tricked-out by the compiler to avoid unnecessary increments and decrements. The principles apply, though.] Running the above code fragment in a different implementation, in which garbage collection is deferred, would *not* close the file descriptor, and the system would potentially run out (depending on when a gc occurred, and/or whether the system would attempt gc on running out of file descriptors, in the hope that the gc would free some up). The subprocess module does go through a bunch of extra work to make sure that any as-yet-uncollected fork()ed processes are eventually waitpid()-ed for. >Can anyone explain the treatment of the pipe FDs opened in the parent >by Popen() to me or point me to some documentation? The best documentation seems generally to be the source. Fortunately subprocess.py is written in Python. (Inspecting C modules is less straightforward. :-) ) >Also, does Popen.returncode contain only the child's exit code or is >does it also contain signal info like the return of os.wait()? >Documentation on this is also unclear to me. "A negative value -N indicates that the child was terminated by signal N (Unix only)." Again, the Python source is handy: def _handle_exitstatus(self, sts): if os.WIFSIGNALED(sts): self.returncode = -os.WTERMSIG(sts) elif os.WIFEXITED(sts): self.returncode = os.WEXITSTATUS(sts) else: # Should never happen raise RuntimeError("Unknown child exit status!") The only things left out are the core-dump flag, and stopped/suspended. The latter should never occur as os.waitpid() is called with only os.WNOHANG, not os.WUNTRACED (of course a process being traced, stopping at a breakpoint, would mess this up, but subprocess.Popen is not a debugger :-) ). It might be nice to capture os.WCOREDUMPED(sts), though. Also, while I was writing this, I discovered that appears to be a buglet in _cleanup(), with regard to "abandoned" Unix processes that terminate due to a signal. Note that _handle_exitstatus() will set self.returncode to (e.g.) -1 if the child exits due to SIGHUP. The _cleanup() function, however, does this in part: if inst.poll(_deadstate=sys.maxint) >= 0: try: _active.remove(inst) The Unix-specific poll() routine, however, reads: if self.returncode is None: try: pid, sts = os.waitpid(self.pid, os.WNOHANG) if pid == self.pid: self._handle_exitstatus(sts) except os.error: if _deadstate is not None: self.returncode = _deadstate return self.returncode Hence if pid 12345 is abandoned (and thus on _active), and we os.waitpid(12345, os.WNOHANG) and get a status that has a termination signal, we set self.returncode to -N, and return that. Hence inst.poll returns (e.g.) -1 and we never attempt to remove it from _active. Now that its returncode is not None, though, every later poll() will continue to return -1. It seems it would be better to have _cleanup() read: if inst.poll(_deadstate=sys.maxint) is not None: (Note, this is python 2.5, which is what I have installed on my Mac laptop, where I am writing this at the moment). -- In-Real-Life: Chris Torek, Wind River Systems Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603 email: gmail (figure it out) http://web.torek.net/torek/index.html
-- http://mail.python.org/mailman/listinfo/python-list