Re: subprocess woes
On Sep 15, 2:26 pm, kj no.em...@please.post wrote: I'm trying to write a function, sort_data, that takes as argument the path to a file, and sorts it in place, leaving the last sentinel line in its original position (i.e. at the end). Here's what I have (omitting most error-checking code): def sort_data(path, sentinel='.\n'): tmp_fd, tmp = tempfile.mkstemp() out = os.fdopen(tmp_fd, 'wb') cmd = ['/usr/local/bin/sort', '-t', '\t', '-k1,1', '-k2,2'] p = Popen(cmd, stdin=PIPE, stdout=out) in_ = file(path, 'r') while True: line = in_.next() if line != sentinel: p.stdin.write(line) else: break in_.close() p.stdin.close() retcode = p.wait() if retcode != 0: raise CalledProcessError(retcode, cmd) out.write(sentinel) out.close() shutil.move(tmp, path) This works OK, except that it does not catch the stderr from the called sort process. The problem is how to do this. I want to to avoid having to create a new file just to capture this stderr output. I would like instead to capture it to an in-memory buffer. Therefore I tried using a StringIO object as the stderr parameter to Popen, but this resulted in the error StringIO instance has no attribute 'fileno'. How can I capture stderr in the scenario depicted above? TIA! kynn According to the docs for subprocess module (which you don't appear to be using even though that's what you used for your subject line), you can set stderr to stdout: http://docs.python.org/library/subprocess.html You can use cStringIO to create a file-like object in memory: http://docs.python.org/library/stringio.html --- Mike Driscoll Blog: http://blog.pythonlibrary.org -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess woes
Mike Driscoll wrote: You can use cStringIO to create a file-like object in memory: http://docs.python.org/library/stringio.html No, you can't with subprocess. The underlying operating system API requires a file descriptor of a real file. Christian -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess woes
In d87065db-f51f-4afe-924c-f9e4a1eb0...@g23g2000vbr.googlegroups.com Mike Driscoll kyoso...@gmail.com writes: On Sep 15, 2:26=A0pm, kj no.em...@please.post wrote: I'm trying to write a function, sort_data, that takes as argument the path to a file, and sorts it in place, leaving the last sentinel line in its original position (i.e. at the end). =A0Here's what I have (omitting most error-checking code): def sort_data(path, sentinel=3D'.\n'): =A0 =A0 tmp_fd, tmp =3D tempfile.mkstemp() =A0 =A0 out =3D os.fdopen(tmp_fd, 'wb') =A0 =A0 cmd =3D ['/usr/local/bin/sort', '-t', '\t', '-k1,1', '-k2,2'] =A0 =A0 p =3D Popen(cmd, stdin=3DPIPE, stdout=3Dout) =A0 =A0 in_ =3D file(path, 'r') =A0 =A0 while True: =A0 =A0 =A0 =A0 line =3D in_.next() =A0 =A0 =A0 =A0 if line !=3D sentinel: =A0 =A0 =A0 =A0 =A0 =A0 p.stdin.write(line) =A0 =A0 =A0 =A0 else: =A0 =A0 =A0 =A0 =A0 =A0 break =A0 =A0 in_.close() =A0 =A0 p.stdin.close() =A0 =A0 retcode =3D p.wait() =A0 =A0 if retcode !=3D 0: =A0 =A0 =A0 =A0 raise CalledProcessError(retcode, cmd) =A0 =A0 out.write(sentinel) =A0 =A0 out.close() =A0 =A0 shutil.move(tmp, path) This works OK, except that it does not catch the stderr from the called sort process. =A0The problem is how to do this. =A0I want to to avoid having to create a new file just to capture this stderr output. =A0I would like instead to capture it to an in-memory buffer. Therefore I tried using a StringIO object as the stderr parameter to Popen, but this resulted in the error StringIO instance has no attribute 'fileno'. How can I capture stderr in the scenario depicted above? TIA! kynn According to the docs for subprocess module (which you don't appear to be using even though that's what you used for your subject line), Sorry, I should have been clearer. I *am* using subprocess; that's were Popen and PIPE come from. I omitted the import lines along with much else to keep the code concise. Maybe I overdid it. you can set stderr to stdout: This won't do: I'm already using stdout to collect the output of sort. You can use cStringIO to create a file-like object in memory: Nope. I get the same error I get when I try this idea using StringIO (i.e. no attribute 'fileno'). kj -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess woes
Upon re-reading my post I realize that I left out some important details. In h8oppp$qo...@reader1.panix.com kj no.em...@please.post writes: I'm trying to write a function, sort_data, that takes as argument the path to a file, and sorts it in place, leaving the last sentinel line in its original position (i.e. at the end). I neglected to mention that the files I intend to use this with are huge (of the order of 1GB); this is why I want to farm the work out to GNU's sort. Here's what I have (omitting most error-checking code): I should have included the following in the quoted code: from subprocess import Popen, PIPE, CalledProcessError def sort_data(path, sentinel='.\n'): tmp_fd, tmp = tempfile.mkstemp() out = os.fdopen(tmp_fd, 'wb') cmd = ['/usr/local/bin/sort', '-t', '\t', '-k1,1', '-k2,2'] p = Popen(cmd, stdin=PIPE, stdout=out) in_ = file(path, 'r') while True: line = in_.next() if line != sentinel: p.stdin.write(line) else: break in_.close() p.stdin.close() retcode = p.wait() if retcode != 0: raise CalledProcessError(retcode, cmd) out.write(sentinel) out.close() shutil.move(tmp, path) This works OK, except that it does not catch the stderr from the called sort process. The problem is how to do this. I want to to avoid having to create a new file just to capture this stderr output. I would like instead to capture it to an in-memory buffer. Therefore I tried using a StringIO object as the stderr parameter to Popen, but this resulted in the error StringIO instance has no attribute 'fileno'. How can I capture stderr in the scenario depicted above? TIA! kynn -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess woes
On Tue, Sep 15, 2009 at 12:26 PM, kj no.em...@please.post wrote: I'm trying to write a function, sort_data, that takes as argument the path to a file, and sorts it in place, leaving the last sentinel line in its original position (i.e. at the end). Here's what I have (omitting most error-checking code): def sort_data(path, sentinel='.\n'): tmp_fd, tmp = tempfile.mkstemp() out = os.fdopen(tmp_fd, 'wb') cmd = ['/usr/local/bin/sort', '-t', '\t', '-k1,1', '-k2,2'] p = Popen(cmd, stdin=PIPE, stdout=out) in_ = file(path, 'r') while True: line = in_.next() if line != sentinel: p.stdin.write(line) else: break in_.close() p.stdin.close() retcode = p.wait() if retcode != 0: raise CalledProcessError(retcode, cmd) out.write(sentinel) out.close() shutil.move(tmp, path) This works OK, except that it does not catch the stderr from the called sort process. The problem is how to do this. I want to to avoid having to create a new file just to capture this stderr output. I would like instead to capture it to an in-memory buffer. Therefore I tried using a StringIO object as the stderr parameter to Popen, but this resulted in the error StringIO instance has no attribute 'fileno'. How can I capture stderr in the scenario depicted above? Use a pipe by setting stderr=PIPE?: p = Popen(cmd, stdin=PIPE, stdout=out, stderr=PIPE) #... error_output = p.stderr.read() Or am I missing something? Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess woes
In mailman.1498.1253057406.2854.python-l...@python.org Chris Rebert c...@rebertia.com writes: On Tue, Sep 15, 2009 at 12:26 PM, kj no.em...@please.post wrote: I'm trying to write a function, sort_data, that takes as argument the path to a file, and sorts it in place, leaving the last sentinel line in its original position (i.e. at the end). =C2=A0Here's what I have (omitting most error-checking code): def sort_data(path, sentinel=3D'.\n'): =C2=A0 =C2=A0tmp_fd, tmp =3D tempfile.mkstemp() =C2=A0 =C2=A0out =3D os.fdopen(tmp_fd, 'wb') =C2=A0 =C2=A0cmd =3D ['/usr/local/bin/sort', '-t', '\t', '-k1,1', '-k2,2'= ] =C2=A0 =C2=A0p =3D Popen(cmd, stdin=3DPIPE, stdout=3Dout) =C2=A0 =C2=A0in_ =3D file(path, 'r') =C2=A0 =C2=A0while True: =C2=A0 =C2=A0 =C2=A0 =C2=A0line =3D in_.next() =C2=A0 =C2=A0 =C2=A0 =C2=A0if line !=3D sentinel: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0p.stdin.write(line) =C2=A0 =C2=A0 =C2=A0 =C2=A0else: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0break =C2=A0 =C2=A0in_.close() =C2=A0 =C2=A0p.stdin.close() =C2=A0 =C2=A0retcode =3D p.wait() =C2=A0 =C2=A0if retcode !=3D 0: =C2=A0 =C2=A0 =C2=A0 =C2=A0raise CalledProcessError(retcode, cmd) =C2=A0 =C2=A0out.write(sentinel) =C2=A0 =C2=A0out.close() =C2=A0 =C2=A0shutil.move(tmp, path) This works OK, except that it does not catch the stderr from the called sort process. =C2=A0The problem is how to do this. =C2=A0I want to= to avoid having to create a new file just to capture this stderr output. =C2=A0I would like instead to capture it to an in-memory buffer. Therefore I tried using a StringIO object as the stderr parameter to Popen, but this resulted in the error StringIO instance has no attribute 'fileno'. How can I capture stderr in the scenario depicted above? Use a pipe by setting stderr=3DPIPE?: p =3D Popen(cmd, stdin=3DPIPE, stdout=3Dout, stderr=3DPIPE) #... error_output =3D p.stderr.read() Thanks, that did the trick. Thanks! kj -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess woes
Hi Dennis, That works great ! thanks for the correction regards, KM-- On 8/29/06, Dennis Lee Bieber [EMAIL PROTECTED] wrote: On Tue, 29 Aug 2006 18:17:47 +0530, km [EMAIL PROTECTED] declaimed the following in comp.lang.python: ##code start ## import subprocess as sp x = 'GSQIPSHYWKKNLWYYSHEIDGGCHNMW' p0 = sp.Popen([echo,x], stdout=sp.PIPE ) Why use this at all? p1 = sp.Popen([fasta34,-q,@,s],stdin=p0.stdout, stdout=sp.PIPE) output = p1.communicate()[0] Just feed x to this directly... (untested): p1 = sp.Popen([fasta34,-q,@,s],stdin=sp.PIPE, stdout=sp.PIPE)output = p1.communicate(x)[0]-- WulfraedDennis Lee Bieber KD6MOG [EMAIL PROTECTED] [EMAIL PROTECTED] HTTP://wlfraed.home.netcom.com/ (Bestiaria Support Staff: [EMAIL PROTECTED]) HTTP://www.bestiaria.com/-- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess woes
Hi Dennis, That works great. thanks for the correction. The 'output' variablehas the returneddata as string obbject. how can i get it as a list object with elements as line by line? Is it that p1.communicate()[0] by default returns a singlestring only ? regards, KM On 8/29/06, Dennis Lee Bieber [EMAIL PROTECTED] wrote: On Tue, 29 Aug 2006 18:17:47 +0530, km [EMAIL PROTECTED] declaimed the following in comp.lang.python: ##code start ## import subprocess as sp x = 'GSQIPSHYWKKNLWYYSHEIDGGCHNMW' p0 = sp.Popen([echo,x], stdout=sp.PIPE ) Why use this at all? p1 = sp.Popen([fasta34,-q,@,s],stdin=p0.stdout, stdout=sp.PIPE) output = p1.communicate()[0] Just feed x to this directly... (untested): p1 = sp.Popen([fasta34,-q,@,s],stdin=sp.PIPE, stdout=sp.PIPE)output = p1.communicate(x)[0]-- WulfraedDennis Lee Bieber KD6MOG [EMAIL PROTECTED] [EMAIL PROTECTED] HTTP://wlfraed.home.netcom.com/ (Bestiaria Support Staff: [EMAIL PROTECTED]) HTTP://www.bestiaria.com/-- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess woes
Dennis Lee Bieber wrote: On Tue, 29 Aug 2006 18:17:47 +0530, km [EMAIL PROTECTED] declaimed the following in comp.lang.python: ##code start ## import subprocess as sp x = 'GSQIPSHYWKKNLWYYSHEIDGGCHNMW' p0 = sp.Popen([echo,x], stdout=sp.PIPE) Why use this at all? p1 = sp.Popen([fasta34,-q,@,s],stdin=p0.stdout, stdout=sp.PIPE) output = p1.communicate()[0] Just feed x to this directly... (untested): p1 = sp.Popen([fasta34,-q,@,s],stdin=sp.PIPE, stdout=sp.PIPE) output = p1.communicate(x)[0] -- WulfraedDennis Lee Bieber KD6MOG [EMAIL PROTECTED] [EMAIL PROTECTED] HTTP://wlfraed.home.netcom.com/ (Bestiaria Support Staff: [EMAIL PROTECTED]) HTTP://www.bestiaria.com/ You can also pass data to a subprocess like this: p1 = sp.Popen([fasta34,-q,@,s],stdin=sp.PIPE, stdout=sp.PIPE) p1.stdin.write(x) p1.stdin.close() But I think communicate() would be better for you in this case. Peace, ~Simon -- http://mail.python.org/mailman/listinfo/python-list
