Re: subprocess.Popen instance hangs
On Thu, 29 Aug 2013 17:00:21 -0800, Tim Johnson wrote: ## This appears to be what works. def __exec(self,args) : Run the process with arguments p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE) while 1 : output = p.stdout.read() If the process tries to write more than a pipe's worth of data to stderr, before closing stdout, it will block indefinitely. If you want to process both stdout and stderr, you have to be able to consume the data in whatever order the process generates it, which means either using multiple threads or (on Unix) select/poll or non-blocking I/O. This is what the .communicate() method does (threads on Windows, select/poll on Unix). The alternative is to merge both streams with stderr=subprocess.STDOUT, or redirect one of them to a file (or /dev/null, etc). -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess.Popen instance hangs
On Fri, Aug 30, 2013 at 11:32 AM, Tim Johnson t...@akwebsoft.com wrote: The objective is to display all output, but to also separate error messages from normal output. I still think you want to use communicate(). Like this: p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE) output, err = p.communicate() That's it. No need for a loop, or manually handling the fact that stderr and/or stdout could end up with a full buffer and start to block. -- Jerry -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess.Popen instance hangs
* Nobody nob...@nowhere.com [130830 06:55]: On Thu, 29 Aug 2013 17:00:21 -0800, Tim Johnson wrote: ## This appears to be what works. def __exec(self,args) : Run the process with arguments p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE) while 1 : output = p.stdout.read() If the process tries to write more than a pipe's worth of data to stderr, before closing stdout, it will block indefinitely. If you want to process both stdout and stderr, you have to be able to consume the data in whatever order the process generates it, which means either using multiple threads or (on Unix) select/poll or non-blocking I/O. This is what the .communicate() method does (threads on Windows, select/poll on Unix). The alternative is to merge both streams with stderr=subprocess.STDOUT, or redirect one of them to a file (or /dev/null, etc). In earlier code I, I was merging them... :) Like I said: gnarly! What if I were to do something like: ## code while 1: output = p.stout.read() err = p.stderr.read() ## trapping for AttributeError, etc.. ## /code break'ing if either no output or value in `err' ?? The objective is to display all output, but to also separate error messages from normal output. thank you -- Tim tim at tee jay forty nine dot com or akwebsoft dot com http://www.akwebsoft.com -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess.Popen instance hangs
* Jerry Hill malaclyp...@gmail.com [130830 07:48]: On Fri, Aug 30, 2013 at 11:32 AM, Tim Johnson t...@akwebsoft.com wrote: The objective is to display all output, but to also separate error messages from normal output. I still think you want to use communicate(). Like this: p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE) output, err = p.communicate() That's it. No need for a loop, or manually handling the fact that stderr and/or stdout could end up with a full buffer and start to block. The following code : p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE) errmsg,output = p.communicate() ... Hangs this code : p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE) while 1 : output = p.stdout.read() if output : print(output) else : break ... works -- Tim tim at tee jay forty nine dot com or akwebsoft dot com http://www.akwebsoft.com -- http://mail.python.org/mailman/listinfo/python-list
subprocess.Popen instance hangs
using Python 2.7.1 on OS X 10.7.5 I'm managing a process of drush using an instance of subprocess.Popen The process has a '--verbose' option. When that option is passed as part of the initializer `args' argument, the process will hang. It should be no surprise as drush output with the --verbose option can be _extremely_ verbose, and I can do without it, but I would like to learn how to handle it. I've googled this topic, but my poor little brain is yet to sort out all of the content found. ## my relevant code follows : p = subprocess.Popen(args,stderr=subprocess.STDOUT,stdout=subprocess.PIPE) ## wait() is 'forever' if '--verbose' used exit_status = p.wait() output = p.stdout.read() ## done I 'suspect' that using a tempfile may be the solution, if so, I could use some examples. thanks -- Tim tim at tee jay forty nine dot com or akwebsoft dot com http://www.akwebsoft.com -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess.Popen instance hangs
On 29/08/2013 19:34, Tim Johnson wrote: using Python 2.7.1 on OS X 10.7.5 I'm managing a process of drush using an instance of subprocess.Popen The process has a '--verbose' option. When that option is passed as part of the initializer `args' argument, the process will hang. It should be no surprise as drush output with the --verbose option can be _extremely_ verbose, and I can do without it, but I would like to learn how to handle it. I've googled this topic, but my poor little brain is yet to sort out all of the content found. ## my relevant code follows : p = subprocess.Popen(args,stderr=subprocess.STDOUT,stdout=subprocess.PIPE) ## wait() is 'forever' if '--verbose' used exit_status = p.wait() output = p.stdout.read() ## done I 'suspect' that using a tempfile may be the solution, if so, I could use some examples. The subprocess will terminate when it has finished writing its output, but because you're not consuming any of the output (you're waiting for it to finish), the buffer fills up and blocks the subprocess. Try reading the output or using the .communicate method. Alternatively, pass an open file as the stdout argument. -- http://mail.python.org/mailman/listinfo/python-list
RE: subprocess.Popen instance hangs
Tim Johnson using Python 2.7.1 on OS X 10.7.5 I'm managing a process of drush using an instance of subprocess.Popen The process has a '--verbose' option. When that option is passed as part of the initializer `args' argument, the process will hang. It should be no surprise as drush output with the --verbose option can be _extremely_ verbose, and I can do without it, but I would like to learn how to handle it. I've googled this topic, but my poor little brain is yet to sort out all of the content found. ## my relevant code follows : p = subprocess.Popen(args,stderr=subprocess.STDOUT,stdout=subprocess.PIPE) ## wait() is 'forever' if '--verbose' used exit_status = p.wait() output = p.stdout.read() ## done I 'suspect' that using a tempfile may be the solution, if so, I could use some examples. thanks -- Tim tim at tee jay forty nine dot com or akwebsoft dot com http://www.akwebsoft.com -- I think the documentation covers your issue. Popen.wait() Wait for child process to terminate. Set and return returncode attribute. Warning: will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use [Popen.]communicate() to avoid that. From http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait Does switching to `p.communicate()` solve your problem? ~Ramit This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email. -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess.Popen instance hangs
On Thursday 2013 August 29 11:34, Tim Johnson wrote: using Python 2.7.1 on OS X 10.7.5 I'm managing a process of drush using an instance of subprocess.Popen The process has a '--verbose' option. When that option is passed as part of the initializer `args' argument, the process will hang. It should be no surprise as drush output with the --verbose option can be _extremely_ verbose, and I can do without it, but I would like to learn how to handle it. I've googled this topic, but my poor little brain is yet to sort out all of the content found. ## my relevant code follows : p = subprocess.Popen(args,stderr=subprocess.STDOUT,stdout=subprocess.PIPE) ## wait() is 'forever' if '--verbose' used exit_status = p.wait() output = p.stdout.read() ## done p = subprocess.Popen(args,stderr=subprocess.STDOUT,stdout=subprocess.PIPE) for line in p.stdout: do_something(line) p.wait() -- Yonder nor sorghum stenches shut ladle gulls stopper torque wet strainers. -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess.Popen instance hangs
* MRAB pyt...@mrabarnett.plus.com [130829 11:04]:
On 29/08/2013 19:34, Tim Johnson wrote:
could use some examples.
The subprocess will terminate when it has finished writing its output,
but because you're not consuming any of the output (you're waiting for
it to finish), the buffer fills up and blocks the subprocess.
Try reading the output or using the .communicate method.
Alternatively, pass an open file as the stdout argument.
Kudos to all for the replies. Here is some code to review:
## execute process and read output
p = subprocess.Popen(args,stderr=subprocess.STDOUT,stdout=subprocess.PIPE)
while 1 :
output = p.stdout.read()
if output :
print(output)
else : break
## Check for errors
exit_status = p.wait()
if exit_status :
if p.stderr :
self.__err('Process terminated with exit status: %s' %
(str(exit_status)),
'Following error message was found:'
p.stderr.read())
else :
self.__err('Process terminated with exit status: %s' %
(str(exit_status)))
Without any error from the drush process, this works fine.
I can't quite figure out how to simulate/cause an error from drush,
so I would welcome comments on the error handling
thanks again
--
Tim
tim at tee jay forty nine dot com or akwebsoft dot com
http://www.akwebsoft.com
--
http://mail.python.org/mailman/listinfo/python-list
Re: subprocess.Popen instance hangs
* Tim Johnson t...@akwebsoft.com [130829 10:51]:
using Python 2.7.1 on OS X 10.7.5
I'm managing a process of drush using an instance of subprocess.Popen
...
## This appears to be what works.
def __exec(self,args) :
Run the process with arguments
p = subprocess.Popen(args,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
while 1 :
output = p.stdout.read()
if output :
print(output)
else : break
errmsg = p.communicate()[1]
if errmsg :
self.__err('Process terminated with error:',errmsg)
## Thanks again, gnarly one for me. I am eternally a noob!
--
Tim
tim at tee jay forty nine dot com or akwebsoft dot com
http://www.akwebsoft.com
--
http://mail.python.org/mailman/listinfo/python-list
