Keywords: subprocess stdout stderr unbuffered pty tty pexpect flush setvbuf
I'm trying to find a solution to <URL:http://bugs.python.org/issue1241>. In short: unless specifically told not to, normal C stdio will use full output buffering when connected to a pipe. It will use default (typically unbuffered) output when connected to a tty/pty. This is why subprocess.Popen() won't work with the following program when stdout and stderr are pipes: #include <stdio.h> #include <unistd.h> int main() { int i; for(i = 0; i < 5; i++) { printf("stdout ding %d\n", i); fprintf(stderr, "stderr ding %d\n", i); sleep(1); } return 0; } Then subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) is read using polling, but the pipes don't return any data until the program is done. As expected, specifying a bufsize of 0 or 1 in the Popen() call has no effect. See end of mail for example Python scripts. Unfortunately, changing the child program to flush stdout/err or specifying setvbuf(stdout,0,_IONBF,0); is an undesired workaround in this case. I went with pexpect and it works well, except that I must handle stdout and stderr separately. There seems to be no way to do this with pexpect, or am I mistaken? Are there alternative ways of solving this? Perhaps some way of I'm on Linux, and portability to other platforms is not a requirement. s #Test script using subprocess: import subprocess cmd = '/tmp/slow' # The C program shown above print 'Starting...' proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) while True: lineout = proc.stdout.readline() lineerr = proc.stderr.readline() exitcode = proc.poll() if (not lineout and not lineerr) and exitcode is not None: break if not lineout == '': print lineout.strip() if not lineerr == '': print 'ERR: ' + lineerr.strip() print 'Done.' #Test script using pexpect, merging stdout and stderr: import sys import pexpect cmd = '/home/sveniu/dev/logfetch/bin/slow' print 'Starting...' child = pexpect.spawn(cmd, timeout=100, maxread=1, logfile=sys.stdout) try: while True: child.expect('\n') except pexpect.EOF: pass print 'Done.' -- http://mail.python.org/mailman/listinfo/python-list