Martin Walsh wrote: > Eric Brunson wrote: > >> I'm coming in late to the discussion and thought that someone would >> explain it succinctly, but there have been so many correct statements >> which I feel fail to nail down the problem that I thought I'd chime in. >> >> > > Hi Eric, > > Thank you for your considerate response. Your analysis is spot on -- > speaking for myself, of course -- I'm mostly just flailing here; not > having a very strong grasp of how all the components work > together (kernel, shell, python, and the subprocess module). But I'm > really enjoying this thread and learning a lot in the 'process' ... > ahem, sorry for the pun ... painful :) > > >> Here's the important concepts to understand. The pipe is a construct of >> the shell, you are spawning a shell to run your entire command line. >> When the "make" finishes and stops accepting the input from "yes", the >> kernel sends a SIGPIPE to the parent process, i.e. the shell. The >> shell's default behavior when receiving the SIGPIPE is to print an error >> message. >> > > You're right. Considering the shell is definitely important and I missed > it. Yet given the following, slightly adapted from the subprocess module > docs (your reference link below) ... > > from subprocess import Popen, PIPE > p1 = Popen(["yes", "Hello"], stdout=PIPE) > p2 = Popen(["head", "-n", "10"], stdin=p1.stdout, stdout=PIPE) > output = p2.communicate()[0] > > ... still produces ... > > yes: standard output: Broken pipe > yes: write error >
Looks like the "yes" command is getting the SIGPIPE and not the shell, as I originally thought. I must have glossed over the "yes:" at the beginning. > If run as a script, the message above shows up in the terminal after the > python script terminates (or apparently so; shell prompt first, then > error). And when using the interactive interpreter... nothing until > exiting the interpreter, when the message again appears in the terminal. > Perhaps this is some peculiarity of my environment, or more likely I am > still missing something. > > >> Your problem, as I interpret it, is that you don't like seeing the error >> message. So you have two choices: 1) find some way to filter it, >> because all it is is a message, or 2) get rid of the shell and handle >> the management of the subprocesses yourself. It's been discussed how to >> suppress the output and it's been discussed how to intercept the signal, >> but I don't thing that anyone has pointed to the definitive python >> construct to handle it properly. >> >> You've had several responses that correctly tell you how to handle a >> signal, but without any testing I'm pretty sure that none of that will >> do you any good with the subprocess.call() construct you're using. The >> shell is your subprocess and it is already handling the signal, so you >> should never see it. The trick is to spawn both processes via python >> *without* an intervening shell to interfere. Here's the canonical >> example: http://docs.python.org/lib/node536.html. You can spawn each >> subprocess, 'yes' and 'make' without the shell being required to pipe >> the output of one to the other, you can do it all completely in python. >> > > Based on my own tests, I don't think it makes a difference. I suspect > the python override for sigpipe is inherited by all child processes, > shell or otherwise, and the SIG_IGN is allowing 'yes' to see the broken > pipe, report it and terminate -- where the default action would normally > be to terminate only. This is, of course, not that far from wild > speculation. Though, I am curious if others observe the same behavior. > I'd have to break out some documentation to review signal mask inheritance, it's been so long it escapes me. > Then again, the error is *not* displayed for any of tests where > signal.signal(signal.SIGPIPE, signal.SIG_DFL) is added, in either the > global scope or in the form of a preexec_fn argument passed to > subprocess, with shell=True or without. > > IMHO, this is certainly a tripping point if true. Granted, probably a > fringe case and relatively rare since it requires spawning a subprocess, > which writes to a pipe that is broken before the subprocess ends, and is > not implemented to handle the event gracefully. But I still find it > weirdly intriguing. > Since "yes" is designed to spit out its output indefinitely, it seems like a bad behavior to print a message on SIGPIPE. It's always going to happen, it's the expected result. The versions of "yes" on my linux box here and a solaris box at work don't seem to elicit the same error message. Just another reason why I would look for a solution that would avoid the "yes" altogether. :-) >> Again, sorry to come in late, but while reading many absolutely factual >> > > <off-topic-blather> > As far as I'm concerned, no apologies are necessary -- not that you are > apologizing to me necessarily :) But nonetheless, thank you for helping. > > In fact, a big thank you to all the tutors. I've been a member of the > list for quite a few years now, reaping the benefits of your collective > experience (like many others I would imagine), mostly by lurking. You > provide a tremendous service to the python community, especially those > new to python or programming in general, with near super-human patience > and skill. Quite a feat! FWIW, thanks. > </off-topic-blather> > Even being one of the more vocal on the list, I'll add my vote to your commendations. I learn just as much on the list as I feel I teach, it's a great forum with some great participants. > Marty > > >> an correct responses, they all seemed to be either handling the output >> or discussing signal handling without, in my mind, pointing out that >> it's the shell that's giving you the problems. I think it's Python's >> default behavior to ignore SIGPIPE, as Martin comments on in his latest >> reply. It's just that the shell has already accepted and handled the >> signal. >> >> Interestingly, when you eliminate the shell and use something similar to >> the example code from above, you can pretty easily see how to get rid of >> the 'yes' and just feed the make subprocess input yourself, further >> simplifying the code. >> >> I'll admit, I've stated all this without actually running any test, so >> please, if anyone can show differently, I'm happy to have my >> understanding corrected. >> >> I hope that helps, not only with your specific problem, but from a "big >> picture" standpoint, also. Nothing I hate more than writing code that I >> don't really understand. :-) >> >> e. >> >> James wrote: >> >>> Hi, >>> >>> I have a snippet of code in a Python script I'm whipping up that's >>> causing a not-so-pretty output. Here's the code: >>> >>> subprocess.call( "yes '' | make oldconfig" , shell=True ) >>> >>> When I run this code, Python loyally executes the command, and then I >>> see the following error on my console: >>> >>> ----- >>> >>> yes: standard output: Broken pipe >>> yes: write error >>> >>> ----- >>> >>> I did some Googling and I believe found the reason for this error >>> ("yes" executes forever, and complains when Python kills the process >>> off). However, I'd like to figure out a way to get rid of the error >>> (or hide it) so that it's not visible to the person running the >>> script (it's not the prettiest thing to see scroll by your screen :)). >>> >>> Thoughts / ideas? >>> >>> Thanks! >>> .james >>> _______________________________________________ >>> Tutor maillist - Tutor@python.org >>> http://mail.python.org/mailman/listinfo/tutor >>> >>> >> _______________________________________________ >> Tutor maillist - Tutor@python.org >> http://mail.python.org/mailman/listinfo/tutor >> > > > _______________________________________________ > Tutor maillist - Tutor@python.org > http://mail.python.org/mailman/listinfo/tutor > _______________________________________________ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
