Re: [Tutor] execute an OS command, get the output
I gotta admit, this took me by surprise, too, but my guess is that once the head command is done, it closes the pipe it's reading from, which is being filled by grep; grep takes the hint and terminates, closing the pipe it's reading from, which is being filled by strings; and strings takes the hint and terminates, even though it hasn't gotten through the entire file. yeah, that kind of makes sense. I forgot you were using head not tail - I normally use tail for that kind of thing - in fact I think I've only used head about 2 or 3 times in my life! But head could well stop as soon as it gets the first 10 lines from grep. Interesting, I must try some experiments Alan G ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
Just for the heack of it, I wrote a tiny Python echo program, and interposed it in the pipe between the strings and grep command: while 1: line = raw_input() print line The command line now looks like this: strings 3193.DAT | python echo.py | grep Newsgroups: | head -10 Newsgroups: comp.lang.python.announce ... Newsgroups: comp.lang.python.announce Traceback (most recent call last): File echo.py, line 3, in ? print line IOError: [Errno 22] Invalid argument close failed: [Errno 22] Invalid argument Thats exactly what I ghad in mind! :-) You've saved me the effort, thanks agin. Alan G ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
Wes,
I haven't tried this but I notice you don't specify shell=True in your Popen
calls,
that might be the cause - ypou aren't acting like the shell does.
But aws I say thats just a guess based on a quick scim of your port.
Alan G.
- Original Message -
From: w chun [EMAIL PROTECTED]
To: tutor@python.org
Sent: Sunday, March 12, 2006 3:37 AM
Subject: Re: [Tutor] execute an OS command, get the output
ok, while we're on the subject, i thought i should ask a question for
once(!). it's been awhile since i played with pipes, and i ran into
this problem on the same day as this post!
if i'm in the shell and playing with a 2-way game with the 'tr' command like
so:
$ tr '[a-z]' '[A-Z]'
us
US
ca
CA
de
DE
$
i can interactively give the cmd some input via stdin and get
something out of stdout... giving and taking until i press ^D to
terminate the program.
however, when i try this using subprocess/popen2, i find that i can't
do the same interaction, i.e. writing to tr's stdin, reading from tr's
stdout, ad nauseum. it seems that i have to give tr all of my input,
then close tr's stdin pipe in order to read anything. (any attempts
at reading before closing results in a blocked OS call, whether it's
using the FD itself or os.read() on its fileno(). the only way for it
to work as i described above is like this:
p = subprocess.Popen(('tr', '[a-z]', '[A-Z]'),
stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True)
po, pi = p.stdin, p.stdout
po.write('us\n')
po.write('ca\n')
po.write('de\n')
po.close()
pi.readlines()
['US\n', 'CA\n', 'DE\n']
pi.close()
i was hoping to do somethign more on the lines of:
po.write('us\n')
pi.read() # or pi.readline()
'US\n'
po.write('ca\n')
pi.read()
'CA\n'
but to no avail. neither sending po.write(chr(4)) nor po.flush() seem
to work. i basically can tell whether a read() will block (or not)
using select, as in:
from select import select
sel = select([pi], [], [pi], 0)
sel
([], [], []) --- MEANS IT WILL BLOCK
or
([open file 'fdopen', mode 'r' at 0x39ac80], [], []) -- WON'T BLOCK
is there anyway to do more asynchronous I/O where i can more or less
interact with the other procses rather than what it's doing now?
thanks you guys... i love the tutors!
-wesley
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
yup, that's right. turns out that it doesn't really matter whether
the cmd is run in a shell or not -- that's one of the great things
about subprocess -- you get to choose. with os.system(), *everything*
is run through the shell.
apparently, this problem isn't new... the problem is that the other
program doesn't provide unbuffered output. i don't know why it works
in pure shell (and not programming pipes) environment, but it does.
here is someone else that run into it, and they had access to the C
source to the other program to add a flush() call to get the
communication the way they want it (similar to my desired output):
http://www.dalkescientific.com/writings/diary/archive/2005/04/15/wrapping_command_line_programs_II.html
cheers,
-wesley
ps. on Mac OS X, their 'tr' command has a -u (unbuffered output)
version, and once i added that option, my code worked exactly the want
i desired. so i'm only SOL on the Linux one. :-)
On 3/12/06, Alan Gauld [EMAIL PROTECTED] wrote:
Wes,
I haven't tried this but I notice you don't specify shell=True in your Popen
calls,
that might be the cause - ypou aren't acting like the shell does.
But aws I say thats just a guess based on a quick scim of your port.
Alan G.
- Original Message -
From: w chun [EMAIL PROTECTED]
To: tutor@python.org
Sent: Sunday, March 12, 2006 3:37 AM
Subject: Re: [Tutor] execute an OS command, get the output
ok, while we're on the subject, i thought i should ask a question for
once(!). it's been awhile since i played with pipes, and i ran into
this problem on the same day as this post!
if i'm in the shell and playing with a 2-way game with the 'tr' command like
so:
$ tr '[a-z]' '[A-Z]'
us
US
ca
CA
de
DE
$
i can interactively give the cmd some input via stdin and get
something out of stdout... giving and taking until i press ^D to
terminate the program.
however, when i try this using subprocess/popen2, i find that i can't
do the same interaction, i.e. writing to tr's stdin, reading from tr's
stdout, ad nauseum. it seems that i have to give tr all of my input,
then close tr's stdin pipe in order to read anything. (any attempts
at reading before closing results in a blocked OS call, whether it's
using the FD itself or os.read() on its fileno(). the only way for it
to work as i described above is like this:
p = subprocess.Popen(('tr', '[a-z]', '[A-Z]'),
stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True)
po, pi = p.stdin, p.stdout
po.write('us\n')
po.write('ca\n')
po.write('de\n')
po.close()
pi.readlines()
['US\n', 'CA\n', 'DE\n']
pi.close()
i was hoping to do somethign more on the lines of:
po.write('us\n')
pi.read() # or pi.readline()
'US\n'
po.write('ca\n')
pi.read()
'CA\n'
but to no avail. neither sending po.write(chr(4)) nor po.flush() seem
to work. i basically can tell whether a read() will block (or not)
using select, as in:
from select import select
sel = select([pi], [], [pi], 0)
sel
([], [], []) --- MEANS IT WILL BLOCK
or
([open file 'fdopen', mode 'r' at 0x39ac80], [], []) -- WON'T BLOCK
is there anyway to do more asynchronous I/O where i can more or less
interact with the other procses rather than what it's doing now?
thanks you guys... i love the tutors!
-wesley
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Core Python Programming, Prentice Hall, (c)2006,2001
http://corepython.com
wesley.j.chun :: wescpy-at-gmail.com
cyberweb.consulting : silicon valley, ca
http://cyberwebconsulting.com
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
1) the os.system module 2a-d) os.popen, and popen2 popen3 and popen4 3) the popen2 module 4) the subprocess module Take a look at the OS topic in my tutorial. The process control section covers all of these calls and explains their differences. It also points out that the latter is intended to replace all the others and shows examples. #1 is synchronous, which is what I want, but it doesn't seem to have any means to capture stdout. Actually they are all synchronous if the command is not interactive. If the command requires input before it completes then it will wait for Python to supply it if using popen (in the same way that it would from a user if using system). #s 2-4 look like they support capturing stdout, but they seem to be asynchronous, which is at the very least overkill for me. output = popen(command).read() isn't too hard is it? More detail, in case it matters: the command line I need to execute is actually three commands, with pipes between them, i.e., string xxx | grep zz | head -10 Just make sure its all within a string and Python doesn't care. But in this case I'd use Pytthon to search for the string and get the last 10 entries rather than grep/head... Alan G Author of the learn to program web tutor http://www.freenetpages.co.uk/hp/alan.gauld ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
On Sat, 11 Mar 2006, Alan Gauld wrote: Take a look at the OS topic in my tutorial. The process control section covers all of these calls and explains their differences. It also points out that the latter is intended to replace all the others and shows examples. Excellent; I'll check it out. Your tutorials have been vey helpful to me in the past. output = popen(command).read() isn't too hard is it? That's exactly the sort of thing I'm thinking of. string xxx | grep zz | head -10 But in this case I'd use Pytthon to search for the string and get the last 10 entries rather than grep/head... That was my initial thought, but the files are 1-3 gigabytes. The string command takes 10-15 minutes to get through a file. I've been using the pipe sequence manually, so I know it works, and only takes a second or two to run. The Python program is for my own use, so I don't have to worry about whether and where string/grep/head are installed. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
But in this case I'd use Pytthon to search for the string and get the last 10 entries rather than grep/head... That was my initial thought, but the files are 1-3 gigabytes. The string command takes 10-15 minutes to get through a file. I've been using the pipe sequence manually, so I know it works, and only takes a second or two to run. But surely the string command will take just as long to run regardless of whether you use pipes or read the output into Python? Are ytou saying that running strings on its own takes 10 minutes but running it in a pipe takes only a few seconds? What happens if you redirect the strings output to a file instead of a pipe - MS DOS terminal windows are notoriously slow, it could be the display update thats taking the trime... Just a suggestion. But 3GB files will take a while on any system! The Python program is for my own use, so I don't have to worry about whether and where string/grep/head are installed. Whatever works for you! :-) Alan G. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
On Sat, 11 Mar 2006, Alan Gauld wrote: But surely the string command will take just as long to run regardless of whether you use pipes or read the output into Python? Surely it will! Except that it doesn't. Are ytou saying that running strings on its own takes 10 minutes but running it in a pipe takes only a few seconds? Yes, exactly. Not any pipe, though; a pipe that is satisfied and closes down after just the first few hundred lines or so. What happens if you redirect the strings output to a file instead of a pipe I haven't tried directing it to an actual file, but when directing it to nul: it's about 10-15 minutes; while directing it to the pipe is just a few seconds. I gotta admit, this took me by surprise, too, but my guess is that once the head command is done, it closes the pipe it's reading from, which is being filled by grep; grep takes the hint and terminates, closing the pipe it's reading from, which is being filled by strings; and strings takes the hint and terminates, even though it hasn't gotten through the entire file. And that all happens in a second or two. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
On Sat, 11 Mar 2006, Terry Carroll wrote: I gotta admit, this took me by surprise, too, but my guess is that once the head command is done, it closes the pipe it's reading from, which is being filled by grep; grep takes the hint and terminates, closing the pipe it's reading from, which is being filled by strings; and strings takes the hint and terminates, even though it hasn't gotten through the entire file. Just for the heack of it, I wrote a tiny Python echo program, and interposed it in the pipe between the strings and grep command: while 1: line = raw_input() print line The command line now looks like this: strings 3193.DAT | python echo.py | grep Newsgroups: | head (the .DAT file is an Agent newsgroup file; the idea here is that by grepping for the first few lines with Newsgroups:, I can tell what newsgroup the .DAT file is associated with.) I get: strings 3193.DAT | python echo.py | grep Newsgroups: | head -10 Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Newsgroups: comp.lang.python.announce Traceback (most recent call last): File echo.py, line 3, in ? print line IOError: [Errno 22] Invalid argument close failed: [Errno 22] Invalid argument My guess is that the Invalid argument here was a write attempting to write to a closed file, the pipe. The string and grep commands (or IO routines called by them) probably detect this and close gracefully; the end result being that a set of piped commands only lasts as long as the shortest-lived consumer, and the upstream producers shut down when they can no longer write to the pipe. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
On Sat, 11 Mar 2006, Terry Carroll wrote: Just for the heack of it, I wrote a tiny Python echo program, and interposed it in the pipe between the strings and grep command: while 1: line = raw_input() print line The command line now looks like this: strings 3193.DAT | python echo.py | grep Newsgroups: | head Actually, to break it down to its simplest, with the python program: for i in range(1,1): print line, i I get: writeabunch.py | head -5 line 1 line 2 line 3 line 4 line 5 Traceback (most recent call last): File C:\Agent-files\Giganews\writeabunch.py, line 2, in ? print line, i IOError: (0, 'Error') close failed: [Errno 22] Invalid argument Same thing, but a far simpler scenario. Interestingly, if I use a low enough range, say range(1,500), it simply runs to completion; I assume that 500 lines in the form of line n\n fit within the first written buffer. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
ok, while we're on the subject, i thought i should ask a question for
once(!). it's been awhile since i played with pipes, and i ran into
this problem on the same day as this post!
if i'm in the shell and playing with a 2-way game with the 'tr' command like so:
$ tr '[a-z]' '[A-Z]'
us
US
ca
CA
de
DE
$
i can interactively give the cmd some input via stdin and get
something out of stdout... giving and taking until i press ^D to
terminate the program.
however, when i try this using subprocess/popen2, i find that i can't
do the same interaction, i.e. writing to tr's stdin, reading from tr's
stdout, ad nauseum. it seems that i have to give tr all of my input,
then close tr's stdin pipe in order to read anything. (any attempts
at reading before closing results in a blocked OS call, whether it's
using the FD itself or os.read() on its fileno(). the only way for it
to work as i described above is like this:
p = subprocess.Popen(('tr', '[a-z]', '[A-Z]'),
stdin=subprocess.PIPE, stdout=subprocess.PIPE, close_fds=True)
po, pi = p.stdin, p.stdout
po.write('us\n')
po.write('ca\n')
po.write('de\n')
po.close()
pi.readlines()
['US\n', 'CA\n', 'DE\n']
pi.close()
i was hoping to do somethign more on the lines of:
po.write('us\n')
pi.read() # or pi.readline()
'US\n'
po.write('ca\n')
pi.read()
'CA\n'
but to no avail. neither sending po.write(chr(4)) nor po.flush() seem
to work. i basically can tell whether a read() will block (or not)
using select, as in:
from select import select
sel = select([pi], [], [pi], 0)
sel
([], [], []) --- MEANS IT WILL BLOCK
or
([open file 'fdopen', mode 'r' at 0x39ac80], [], []) -- WON'T BLOCK
is there anyway to do more asynchronous I/O where i can more or less
interact with the other procses rather than what it's doing now?
thanks you guys... i love the tutors!
-wesley
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
[Tutor] execute an OS command, get the output
I need to execute a command and capture the stdout output. I'm overwhelmed by the plethora of means that Python offers to do something like this, and don't know which, if any, is most applicable: 1) the os.system module 2a-d) os.popen, and popen2 popen3 and popen4 3) the popen2 module 4) the subprocess module #1 is synchronous, which is what I want, but it doesn't seem to have any means to capture stdout. #s 2-4 look like they support capturing stdout, but they seem to be asynchronous, which is at the very least overkill for me. I would ideally like something like this: abc = executeit(commandline)or executeit(commandline, abc) where, after execution, abc be a string or list that would contain the output of the command line commandline after its completed. I'm not worried too much about syntax, though; I'd be happy to just find some synchronous command-exeutor that gives me the standard output by any means. More detail, in case it matters: the command line I need to execute is actually three commands, with pipes between them, i.e., string xxx | grep zz | head -10 My environment is Python 2.4, Windows/XP (with cygwin installed, hence the availablility of the string, grep and head commands) ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
Hi Terry, abc = executeit(commandline)or executeit(commandline, abc) Looks like you are looking for the commands module, it provides: getstatusoutput( cmd) getoutput( cmd) getstatus( file) Which do exactly what tou want. Take a look at the docs at: http://python.active-venture.com/lib/module-commands.html Hope that helps, Hugo ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] execute an OS command, get the output
On Fri, 10 Mar 2006, [ISO-8859-1] Hugo Gonz?lez Monteverde wrote:
Looks like you are looking for the commands module, it provides:
getstatusoutput( cmd)
getoutput( cmd)
getstatus( file)
Which do exactly what tou want. Take a look at the docs at:
http://python.active-venture.com/lib/module-commands.html
Thanks, Hugo. Unfortunately, it doesn't work for me; for example:
print getoutput(dir)
'{' is not recognized as an internal or external command,
operable program or batch file.
The docs say:
Availability: Unix.
And I'm on Windows/XP, so I'm guessing that although the module's there,
it just doesn't work on Windows.
___
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
