Re: [Tutor] pipes and redirecting

2014-05-28 Thread Alan Gauld 

On 27/05/14 21:01, Adam Gold wrote:


dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync | pbzip2  1.img.bz2

The first thing I do is break it into two assignments


And that's the start of the problem because it should be three:
The first command, the second command and the output file.


ddIf = shlex.split(dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync)
compress = shlex.split(pbzip2  /home/adam/1.img.bz2)


compress = pbzip2
outfile = open('/home/adam/1.img.bz2','w')

The redirection symbol is not something subprocess can
use as an argument.


p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE)
p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=subprocess.PIPE)


Use the output file here.

p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=outfile)



I think that the '' redirect needs to be dealt with using the
subprocess module as well but I can't quite put the pieces together.
I'd appreciate any guidance.  Thanks.


Alternatively read the output into a variable using communicate but then 
write it out to the file manually at the end.


[You might also be able to use shell=TRUE but that introduces other 
issues. But I don't know whether using shell includes redirection.]


--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.flickr.com/photos/alangauldphotos

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] pipes and redirecting

2014-05-28 Thread Cameron Simpson 

On 27May2014 21:01, Adam Gold a...@gmx.com wrote:

I'm trying to run the following unix command from within Python as
opposed to calling an external Bash script (the reason being I'm doing
it multiple times within a for loop which is running through a list):

dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync | pbzip2  1.img.bz2


First off, one expedient way to do this is to generate a shell script and pipe 
into sh (or sh -uex, my preferred error sensitive invocation).


  p1 = subprocess.Popen([sh, -uex], stdin=PIPE)
  for num in range(1,11):
print(dd if=/home/adam/%d bs=4k conv=noerror,notrunc,sync | pbzip2  
%d.img.bz2,
  % (num, num), file=p1.stdin)
  p1.stdin.close()
  p1.wait()

Any quoting issues aside, this is surprisingly useful. Let the shell do what it 
is good it.


And NOTHING you've said here requires using bash. Use sh and say sh, it is 
very portable and bash is rarely needed for most stuff.


However, I gather beyond expediency, you want to know how to assemble pipelines 
using subprocess anyway. So...



The first thing I do is break it into two assignments (I know this isn't
strictly necessary but it makes the code easier to deal with):

ddIf = shlex.split(dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync)
compress = shlex.split(pbzip2  /home/adam/1.img.bz2)


This is often worth doing regardless. Longer lines are harder to read.


I have looked at the docs here (and the equivalent for Python 3)
https://docs.python.org/2/library/subprocess.html.  I can get a 'simple'
pipe like the following to work:

p1 = subprocess.Popen([ps], stdout=PIPE)
p2 = subprocess.Popen([grep, ssh], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()
output = p2.communicate()[0]


If you don't care about the stdout of p2 (and you don't, based on your 
dd|pbzip2 example above) and you have left p2's stdout alone so that it goes 
to your normal stdout (eg the terminal) then you don't need to waste time with 
.communicate. I almost never use it myself. As the doco says, prone to 
deadlock. I prefer to just do the right thing explicitly myself, as needed.



I then try to adapt it to my example:

p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE)
p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()
output = p2.communicate()[0]

I get the following error:

pbzip2: *ERROR: File [] NOT found!  Skipping...
---
pbzip2: *ERROR: Input file [/home/adam/1.img.bz2] already has a .bz2
extension!  Skipping

I think that the '' redirect needs to be dealt with using the
subprocess module as well but I can't quite put the pieces together.
I'd appreciate any guidance.  Thanks.


It is as you expect. Consider what the shell does with:

  pbzip2  1.img.bz2

It invokes the command pbzip2 (no arguments) with its output attached to the 
file 1.img.bz2.


So first up: stay away form shlex. It does _not_ do what you need.

Shlex knows about shell string quoting. It does not know about redirections.
It is handy for parsing minilanguages on your own concoction where you want to 
be able to quote strings with spaces. It is not a full on shell parser.


So it (may) serve you well for the dd invocation because there are no 
redirections. But for your usage, so would the .split() method on a string, or 
even better: don't you already know the arguments for your dd? Just fill them 
out directly rather than backtracking from a string.


However, your recipe is very close. Change:

  p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=subprocess.PIPE)

into:

  p2 = subprocess.Popen([pbzip2], stdin=p1.stdout, stdout=open(1.img.bz2, 
w))

Because p2 is writing to 1.img.bz2 you don't need to much about with 
.communicate either. No output to collect, no input to supply.


See where that takes you.

Cheers,
Cameron Simpson c...@zip.com.au
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] pipes and redirecting

2014-05-28 Thread Adam Gold 
 On 27/05/14 21:01, Adam Gold wrote:
 
 dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync | pbzip2  1.img.bz2

 The first thing I do is break it into two assignments
 
 And that's the start of the problem because it should be three:
 The first command, the second command and the output file.
 
 ddIf = shlex.split(dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync)
 compress = shlex.split(pbzip2  /home/adam/1.img.bz2)
 
 compress = pbzip2
 outfile = open('/home/adam/1.img.bz2','w')
 
 The redirection symbol is not something subprocess can
 use as an argument.
 
 p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE)
 p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=subprocess.PIPE)
 
 Use the output file here.
 
 p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=outfile)
 
 
 I think that the '' redirect needs to be dealt with using the
 subprocess module as well but I can't quite put the pieces together.
 I'd appreciate any guidance.  Thanks.
 
 Alternatively read the output into a variable using communicate but then
 write it out to the file manually at the end.
 
 [You might also be able to use shell=TRUE but that introduces other
 issues. But I don't know whether using shell includes redirection.]
 
Thanks Alan, yes, I realise now I needed a third assignment to make this
work.

I actually had an exchange with subscriber 'eryksun' yesterday who did a
great job of pointing me in the right direction.  As a bit of a noob, I
think I replied to the individual rather than the list, hence it doesn't
seem to be in the thread.  For the benefit of the archives I append
below eryksun's initial (there was a bit of follow up but nothing too
important) reply to me.
=

Send p2's stdout to a file:

import subprocess
import shlex

ddIf = shlex.split(dd if=/home/adam/1 bs=4k 
   conv=noerror,notrunc,sync)
compress = pbzip2
filename = /home/adam/1.img.bz2

p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE)
with p1.stdout as fin, open(filename, w) as fout:
p2 = subprocess.Popen(compress, stdin=fin, stdout=fout)

ret1 = p1.wait()
ret2 = p2.wait()

Does this work?

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] pipes and redirecting

2014-05-27 Thread Adam Gold 
I'm trying to run the following unix command from within Python as
opposed to calling an external Bash script (the reason being I'm doing
it multiple times within a for loop which is running through a list):

dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync | pbzip2  1.img.bz2

The first thing I do is break it into two assignments (I know this isn't
strictly necessary but it makes the code easier to deal with):

ddIf = shlex.split(dd if=/home/adam/1 bs=4k conv=noerror,notrunc,sync)
compress = shlex.split(pbzip2  /home/adam/1.img.bz2)

I have looked at the docs here (and the equivalent for Python 3)
https://docs.python.org/2/library/subprocess.html.  I can get a 'simple'
pipe like the following to work:

p1 = subprocess.Popen([ps], stdout=PIPE)
p2 = subprocess.Popen([grep, ssh], stdin=p1.stdout,
stdout=subprocess.PIPE)
p1.stdout.close()
output = p2.communicate()[0]


I then try to adapt it to my example:

p1 = subprocess.Popen(ddIf, stdout=subprocess.PIPE)
p2 = subprocess.Popen(compress, stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()
output = p2.communicate()[0]

I get the following error:

pbzip2: *ERROR: File [] NOT found!  Skipping...
---
pbzip2: *ERROR: Input file [/home/adam/1.img.bz2] already has a .bz2
extension!  Skipping

I think that the '' redirect needs to be dealt with using the
subprocess module as well but I can't quite put the pieces together.
I'd appreciate any guidance.  Thanks.

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor