Re: subprocess help
-- Forwarded message --
From: Steven D'Aprano steve+comp.lang.pyt...@pearwood.info
To: python-list@python.org
Cc:
Date: 16 Apr 2014 12:06:16 GMT
Subject: Re: subprocess help
On Wed, 16 Apr 2014 12:47:03 +0200, Влатко Станковиќ wrote:
Hello,
I'm having some sort of 'problem' when using subprocess calls. This is
the code snipet that i am using:
capture_server1 = '''xvfb-run --auto-servernum ... '''
server1_download = subprocess.Popen(shlex.split(capture_server1),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out_s1, err_s1 = server1_download.communicate()
time.sleep(2)
What's the difference between the server1 code (shown above) and the
server2 code (not shown, but identical as far as I can tell)?
You have to identify what files are remaining open. What does the xvfb-
run process do? What are the rest of the arguments?
My guess is that, depending on the arguments, sometimes xvfb-run leaves
files open even after the process terminates. You should monitor the open
files with lsof which is available on most Linux systems. I don't know
how to do that on other operating systems.
My guess is that, depending on the arguments, sometimes xvfb-run leaves
files open even after the process terminates. You should monitor the open
files with lsof which is available on most Linux systems. I don't know
how to do that on other operating systems.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
xvfb-run accepts some parameters and calls CutyCapt with parameters for it
The command is this:
xvfb-run --auto-servernum --server-num=55 --server-args -screen 0,
1024x768x24 {0} --url={1} --private-browsing=on --out={2}
So server1, opens a process with CutyCapt which points to server1
address/url, does its thing and saves the result in out
Server2 in the other hand has a different address/url, different
server-num, and different out
As a workarround i've added close_fds=True, preexec_fn=os.setsid, and
after communicate(), i am doing os.killpg(server1.pid, signal.SIGUSR1)
Although i am not sure if this will work 100% because i have to wait X days
until something crashes
Any ideas are welcomed
P.S. After adding os.killpg, lsof and ps aux dont show more than 4 or 5
xvfb and cutycapt processes while running
Thanks and Regards
--
https://mail.python.org/mailman/listinfo/python-list
subprocess help
Hello, I'm having some sort of 'problem' when using subprocess calls. This is the code snipet that i am using: capture_server1 = '''xvfb-run --auto-servernum ... ''' server1_download = subprocess.Popen(shlex.split(capture_server1),stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out_s1, err_s1 = server1_download.communicate() time.sleep(2) capture_server2 = '''xvfb-run --auto-servernum ''' server2_download = subprocess.Popen(shlex.split(capture_server2), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out_s2, err_s2 = server2_download.communicate() The problem is the following: - The program runs in a loop, where subprocess is called - It runs for X days, sometimes 3 days, sometimes 5 days - After that i get the following exception: File /usr/lib/python2.7/subprocess.py, line 1091, in pipe_cloexec r, w = os.pipe() OSError: [Errno 24] Too many open files How can i reproduce this on a local machine, and how to make sure that i wont have any errors like this? P.S. Version 2.7 is used with this program -- https://mail.python.org/mailman/listinfo/python-list
Re: subprocess help
On Wed, Apr 16, 2014 at 8:47 PM, Влатко Станковиќ l...@linuxmail.org wrote: capture_server1 = '''xvfb-run --auto-servernum ... ''' server1_download = subprocess.Popen(shlex.split(capture_server1) Separate to your actual problem: Is there a reason for splitting like that, rather than simply using a list of separate arguments? That would be a lot safer and easier; no going through the hassles of quoting and splitting. Your exact problem is likely to be due to unclosed files. I don't know enough about .communicate() to know whether it closes everything immediately or not, but it looks like you're progressively opening more and more and more pipes. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: subprocess help
On Wed, 16 Apr 2014 12:47:03 +0200, Влатко Станковиќ wrote: Hello, I'm having some sort of 'problem' when using subprocess calls. This is the code snipet that i am using: capture_server1 = '''xvfb-run --auto-servernum ... ''' server1_download = subprocess.Popen(shlex.split(capture_server1), stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out_s1, err_s1 = server1_download.communicate() time.sleep(2) What's the difference between the server1 code (shown above) and the server2 code (not shown, but identical as far as I can tell)? [...] The problem is the following: - The program runs in a loop, where subprocess is called - It runs for X days, sometimes 3 days, sometimes 5 days - After that i get the following exception: File /usr/lib/python2.7/subprocess.py, line 1091, in pipe_cloexec r, w = os.pipe() OSError: [Errno 24] Too many open files You have to identify what files are remaining open. What does the xvfb- run process do? What are the rest of the arguments? My guess is that, depending on the arguments, sometimes xvfb-run leaves files open even after the process terminates. You should monitor the open files with lsof which is available on most Linux systems. I don't know how to do that on other operating systems. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Background subprocess help?
danshumaker dan.shuma...@comcast.net (d) wrote: d Hi, d I'm trying to do something as simple as this: d sleep 10; mail -s test dans communicate_with_process d which executes immediately because it is backgrounded with . d or more generically in english: d do some long process in the background; send me mail when it's done; allow d me to quit the tool that started launched the long process. This description does not correspond to the command above. First the command is confusing because it uses quotes in a strange way. If the quotes are meant as Python string quotes there is a problem because the inner quotes are not escaped. If they are meant as shell quotes they are completely wrong. So let us assume that the outer quotes are meant at the linguistic level and the inner quotes are shell quotes. Of course using quotes in such an ambiguous way is confusing. Moreover the quotes around test are superfluous. They are only necessary if there are spaces or other strange characters (shell metachars) in the subject argument. So let us assume that the shell command you want to execute is: sleep 10; mail -s test dans communicate_with_process This will not do the whole thing in the background. It will do the first part (sleep 10) and wait for it to complete; after that it will do the mail command in the background. From your description it seems that you want to do the combo in the background. In the shell this would be: (sleep 10; mail -s test dans communicate_with_process) And communicate_with_process is apparently a file; I suppose it is written by the preceding command. If it would be the standard output of the process it would be something like: (sleep 10 | mail -s test dans) of course with sleep 10 replaced by something that generates output. This is also suggested by your mentioning pipeline below. d I've found several examples that are close but none of the permeatations I d try work. The closest examples I've found start sending signals to d processes and/or callbacks and 50+ lines of python code. d I found the cliUtils module which came close.. but no cigar. d I find it hard to believe that a one liner in bash/tcsh has to end up being d many lines in python. That makes me think I'm missing something. d Surely thousands of people have done this before -- start a background d process and want notification when it's done. Sequencing programs is the strength of a shell. It is not Pythons main task, therefore it takes a little more effort for Python. Running a combo in the background means there must be a process that waits for the first program to complete and then starts the second one. In the shell there will be another shell started, because of the () and that will execute the two commands sequentially. So if you want to do it in Python the same should be done. Python just doesn't have a convenient syntax for this like the shell. d Anybody have any suggestions on modules to try or quick subprocess commands d that work? This came close d http://docs.python.org/library/subprocess.html#replacing-shell-pipeline but d no cigar too. The easiest would be to also use the shell to do the sequencing. Why reinvent the wheel? Like p = Popen(sleep 10 | mail -s \test\ dans, shell=True) As you don't want to wait for the thing to complete, the p is almost useless. If you insist on doing it in Python I would recommend a fork to start the new process. I suppose you are on a Posix system as you use after the shell command. import os from subprocess import Popen, PIPE pid = os.fork() if pid == 0: # child code p1 = Popen([sleep, 10], stdout=PIPE) p2 = Popen([mail, -s, test, dans], stdin=p1.stdout) os._exit(0) You could use: os._exit(os.waitpid(p2.pid, 0)[1]) but as you don't wait for the child the actual exit status code is useless. On Windows there is no fork(), therefore you would have to spawn a separate Python program that only does the body of the if. -- Piet van Oostrum p...@cs.uu.nl URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4] Private email: p...@vanoostrum.org -- http://mail.python.org/mailman/listinfo/python-list
Background subprocess help?
Hi, I'm trying to do something as simple as this: sleep 10; mail -s test dans communicate_with_process which executes immediately because it is backgrounded with . or more generically in english: do some long process in the background; send me mail when it's done; allow me to quit the tool that started launched the long process. I've found several examples that are close but none of the permeatations I try work. The closest examples I've found start sending signals to processes and/or callbacks and 50+ lines of python code. I found the cliUtils module which came close.. but no cigar. I find it hard to believe that a one liner in bash/tcsh has to end up being many lines in python. That makes me think I'm missing something. Surely thousands of people have done this before -- start a background process and want notification when it's done. Anybody have any suggestions on modules to try or quick subprocess commands that work? This came close http://docs.python.org/library/subprocess.html#replacing-shell-pipeline but no cigar too. Thank you so much for any help or suggestions. -d -- View this message in context: http://www.nabble.com/Background-subprocess-help--tp23680206p23680206.html Sent from the Python - python-list mailing list archive at Nabble.com. -- http://mail.python.org/mailman/listinfo/python-list
pre subprocess help needed
I need to execute a command shell process obtain the return code and capture stdout from that shell process. I've done this with 2.4 using subprocess. How do I do it with 2.3? -- http://mail.python.org/mailman/listinfo/python-list
Re: pre subprocess help needed
There is a version of subprocess for 2.3. http://www.lysator.liu.se/~astrand/popen5/ http://effbot.org/downloads/#subprocess On Tue, Aug 02, 2005 at 10:05:00AM -0700, chuck wrote: I need to execute a command shell process obtain the return code and capture stdout from that shell process. I've done this with 2.4 using subprocess. How do I do it with 2.3? -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: pre subprocess help needed
You can also use the spawn functions in os, together with the P_WAIT mode. os.spawnlp(os.P_WAIT, 'ls', 'ls') gconfd-martin kde-martin mcop-martin ssh-PhJzdB6333 gpg-bSRhOE ksocket-martin orbit-martin 0 os.spawnlp(os.P_WAIT, 'spam', 'spam') 127 -- http://mail.python.org/mailman/listinfo/python-list
