Re: [Tutor] subprocess check_output

[Alan Gauld via Tutor]

On 20/09/17 22:09, Derek Smith wrote:
> Why does python output this b and newline characters

Because that's what the command is returning.
The b indicates its an array of bytes (as
opposed to a unicode string)

Note the module dpocumentation for check_output() says:

"By default, this function will return the data as encoded
bytes. The actual encoding of the output data may depend
on the command being invoked, so the decoding to text
will often need to be handled at the application level."

>  how can I may it return just what I need, 
> the number 8 w/no special characters? 

You can't but you can convert the bytes into a number
if that's what you want. Just use the int() function.

> I tried str(scratch), but no go, still prints that garbage.

That just converts the sequence of bytes into a
sequence of characters.


> from subprocess import check_output
> scratch = check_output( [ cmdstr, id, pw, do, "select", "count(*)", "from", 
> "libvolumes", "where", "status='Scratch'", "and", "library_name='TS3200'" ] )

You probably don't need to split the query into separate
words, a single string would work just as well.
Or if you want it formatted nicely a few strings:

scratch = check_output( [ cmdstr, id, pw, do,
  "select count(*) from libvolumes",
  "where status='Scratch' and library_name='TS3200'" ] )

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess check_output

[Derek Smith]

Why does python output this b and newline characters and how can I may it 
return just what I need, the number 8 w/no special characters? See below. I 
tried str(scratch), but no go, still prints that garbage.


#!/bin/env python3

cmdstr= "/usr/bin/dsmadmc"
id  = "-id=dereksmith"
pw   = "-password="
do= "-dataonly=yes"


from subprocess import check_output
scratch = check_output( [ cmdstr, id, pw, do, "select", "count(*)", "from", 
"libvolumes", "where", "status='Scratch'", "and", "library_name='TS3200'" ] )
print (scratch)

__OUTPUT__
# ./ts3200_scratchcount_check.py
email sent successfully
b'   8\n'


Thx!!
Derek Smith  |  Unix/TSM Administrator  | Racksquared Data Centers
::  dereksm...@racksquared.com  *: 
www.racksquared.com |  
www.racksquared.jobs

[cid:image003.png@01D2E9AA.1B9CF8F0]

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess module seems to be failing...

[dirkjso...@gmail.com]

Add sound of me beating my head with a bat! :)

Thanks Peter!
-I was a bit too precise in my naming conventions for later review!




On 06/03/2016 02:30 PM, Peter Otten wrote:

dirkjso...@gmail.com wrote:


Running Linux Mint 17.3
Python3.4.3

I'm following a Tutor concerning the subprocess module which said to
open a Terminal SHELL and type:

$ python3
  >>> import subprocess
  >>> subprocess.call('ls', shell=True)

Last night the subprocess.call portion was erroring saying their was no
'call' attribute. I tried asking
dir(subprocess) and call wasn't listed there. I figured I would try
again this morning, however, this
is what I was greeted with shortly after booting up and launching my
Terminal:

[justme@ispy] ~$ python3
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
  >>>
  >>> import subprocess
Traceback (most recent call last):
File "", line 1, in 
File "/home/justme/python_work/subprocess.py", line 39

Look carefully at the line above. You wrote a script and called it
subprocess.py. Bad idea ;)

Delete or rename your subprocess.py, and the one in the standard library
will become visible again and work as smoothly as ever.


  ^
SyntaxError: EOF while scanning triple-quoted string literal
  >>>

Seems like the module is deteriorating or something, as yesterday it
accepted the import command
to include it!  :)

I found a subprocess.py in /usr/lib/python3.4 and grabbed it with gedit
and looked at it line by line. It has tripple double quotes but also
single double quotes throughout it in places, but they all seemed to
have a companion quote to offset them...and the entire text area is one
solid color which would seem to indicate
it is being ignored by python...

What should I do now?

Thanks!


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess module seems to be failing...

[Peter Otten]

dirkjso...@gmail.com wrote:

> Running Linux Mint 17.3
> Python3.4.3
> 
> I'm following a Tutor concerning the subprocess module which said to
> open a Terminal SHELL and type:
> 
> $ python3
>  >>> import subprocess
>  >>> subprocess.call('ls', shell=True)
> 
> Last night the subprocess.call portion was erroring saying their was no
> 'call' attribute. I tried asking
> dir(subprocess) and call wasn't listed there. I figured I would try
> again this morning, however, this
> is what I was greeted with shortly after booting up and launching my
> Terminal:
> 
> [justme@ispy] ~$ python3
> Python 3.4.3 (default, Oct 14 2015, 20:28:29)
> [GCC 4.8.4] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>  >>>
>  >>> import subprocess
> Traceback (most recent call last):
>File "", line 1, in 
>File "/home/justme/python_work/subprocess.py", line 39

Look carefully at the line above. You wrote a script and called it 
subprocess.py. Bad idea ;)

Delete or rename your subprocess.py, and the one in the standard library 
will become visible again and work as smoothly as ever.

> 
>  ^
> SyntaxError: EOF while scanning triple-quoted string literal
>  >>>
> 
> Seems like the module is deteriorating or something, as yesterday it
> accepted the import command
> to include it!  :)
> 
> I found a subprocess.py in /usr/lib/python3.4 and grabbed it with gedit
> and looked at it line by line. It has tripple double quotes but also
> single double quotes throughout it in places, but they all seemed to
> have a companion quote to offset them...and the entire text area is one
> solid color which would seem to indicate
> it is being ignored by python...
> 
> What should I do now?
> 
> Thanks!
> 
> 
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess module seems to be failing...

[dirkjso...@gmail.com]

Running Linux Mint 17.3
Python3.4.3

I'm following a Tutor concerning the subprocess module which said to 
open a Terminal SHELL and type:


$ python3
>>> import subprocess
>>> subprocess.call('ls', shell=True)

Last night the subprocess.call portion was erroring saying their was no 
'call' attribute. I tried asking
dir(subprocess) and call wasn't listed there. I figured I would try 
again this morning, however, this
is what I was greeted with shortly after booting up and launching my 
Terminal:


[justme@ispy] ~$ python3
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import subprocess
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/justme/python_work/subprocess.py", line 39

^
SyntaxError: EOF while scanning triple-quoted string literal
>>>

Seems like the module is deteriorating or something, as yesterday it 
accepted the import command

to include it!  :)

I found a subprocess.py in /usr/lib/python3.4 and grabbed it with gedit 
and looked at it line by line. It has tripple double quotes but also 
single double quotes throughout it in places, but they all seemed to 
have a companion quote to offset them...and the entire text area is one 
solid color which would seem to indicate

it is being ignored by python...

What should I do now?

Thanks!


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess outputing wrong info to command line

[Steven D'Aprano]

On Fri, Feb 20, 2015 at 06:58:17PM -0500, brads wrote:

> My subprocess is in error but the same command at the command line works
> fine.

Are you running the Python script as the same user and from the same 
location as the command line? If you are running the Python script as an 
unprivileged user from your home directory, and the command line as root 
from the directory holding the key, then it is not surprising that they 
will get different results.

Does the dnssec-signzone command use any environment variables? Perhaps 
they are not being inherited by the Python subprocess.

Also, a note about sending code to the list. For some reason, every line 
in your code is separated by blank lines:

> # cat makekeys.py
> 
> #!/usr/bin/python3.4
> 
> import subprocess
> 
> import sys

etc. That makes it very hard to read. Please fix that.

Secondly, you have a huge amount of extraneous code which is irrelevant 
to your problem with subprocess. You calculate dates, ask the user for 
input, delete files, and write new files, calculate arrays and more. 
None of that has anything to do with subprocess. Take it out. Reduce 
your problem to the smallest possible code which shows the problem. 9 
times out of 10, the process of cutting your code down to manageable 
size will actually help you to solve your own problem, and the other 1 
time out of 10 we have a smaller and less confusing piece of code to try 
to understand.

You also have dead, obsolete code commented out. Comments should not be 
used for keeping history in the file, comments should be used for 
commenting about the code. Best practice is to use a revision control 
system like hg, but if you're not up to that at least delete the dead 
code before posting.

When running into a problem with subprocess, your first step should be 
to *exactly* duplicate the successful command:

dnssec-signzone -e2018033000 -p -t -g -k Ktest123.com.ksk.key -o
  test123.com test123.com.external Ktest123.com.zsk.key


So you should try running that from subprocess:

subprocess.call([
'dnssec-signzone', '-e2018033000', '-p', '-t', '-g', 
'-k', 'Ktest123.com.ksk.key', '-o', 'test123.com', 
'test123.com.external', 'Ktest123.com.zsk.key',
])


and see if it still works. If it still does not work, that strongly 
suggests a problem with the environment: you are running it as the wrong 
user, in the wrong location, missing enviromnent variables or 
permissions. If it works in Python, then you can start replacing each 
constant argument with a calculated argument, *one argument at a time*, 
and see where you introduce the bug.



-- 
Steve
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess outputing wrong info to command line

[brads]

My subprocess is in error but the same command at the command line works
fine.

 

 

# cat makekeys.py

#!/usr/bin/python3.4

import subprocess

import sys

import string

import os.path

import datetime

import shlex

from time import gmtime, strftime

from subprocess import Popen, PIPE, STDOUT

 

pretime = strftime("%Y%m%d%H", gmtime())

time = datetime.datetime.strptime(pretime,'%Y%m%d%H')

print (time)

plustime = datetime.timedelta(days=730)

timeadd = (time + plustime)

str(timeadd)

#ndate = datetime.strptime(timeadd, '%Y%m%d%H')

#timeadd = timeadd.replace(tzinfo=UTC())

print (timeadd)

 

dname = input("Enter the domain to configure keys for? ")

if os.path.exists(dname+".external.signed"):

os.remove(dname+".external.signed")

#os.remove(dname+".external")

os.remove(dname+".ksk.key")

os.remove(dname+".zsk.key")

os.remove(dname+".ksk.private")

os.remove(dname+".zsk.private")

fd = open( dname+".external", 'w')

fd.write("$TTL 86400\n")

fd.write("$ORIGIN "+dname+".\n")

fd.write("@  1D  IN SOA yoda.ex-mailer.com.  admin@"+dname+".(\n")

fd.write("  "+strftime("%Y%m%d%H", gmtime())+"\n")

#fd.write(" "+repr(timeadd)+"\n")

fd.write("  3h\n")

fd.write("  1h\n")

fd.write("  1w\n")

fd.write("  1h)\n")

fd.write("  IN NS   yoda.ex-mailer.com.\n")

fd.write("  IN NS   r2d2.ex-mailer.com.\n")

fd.write(dname+".   IN TXT  v=spf1 mx a:r2d2.ex-mailer.com
-all\n")

fd.write(dname+".   MX 0r2d2.ex-mailer.com.\n")

fd.write("mail."+dname+".   IN A107.191.60.48\n")

fd.write("$include /usr/local/etc/namedb/K"+dname+".zsk.key ; ZSK\n")

fd.write("$include /usr/local/etc/namedb/K"+dname+".ksk.key ; KSK\n")

fd.close()

 

 

result = subprocess.check_output(["dnssec-keygen", "-f", "KSK", "-r",
"/dev/urandom", "-a", "RSASHA256", "-b", "2048", "-n", "ZONE", dname])

result_utf8 = result.decode("utf-8").strip()

mylist = list(result_utf8)

print (mylist[0])

listlen= len(mylist)

array = list()

listlen -= 11

i = 0

while( i < listlen ):

#if mylist != '\n' ^ mylist != '':

array.insert(i, mylist[i])

i = i + 1

combined = "".join(array)

print ('combined')

print (combined)

fmove = subprocess.call(["mv", result_utf8+".key",combined +".ksk.key"])

fmove = subprocess.call(["mv", result_utf8+".private",combined
+".ksk.private"])

 

zresult =
subprocess.check_output(["dnssec-keygen","-r","/dev/urandom","-a","RSASHA256
","-b","2048","-n","ZONE", dname])

zresult_utf8 = zresult.decode("utf-8").strip()

myzlist = list(zresult_utf8)

print (myzlist[0])

zlistlen= len(myzlist)

zarray = list()

zlistlen -= 11

zi = 0

while( zi https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess how to use?

[Cameron Simpson]

On 06Nov2014 22:18, jarod...@libero.it  wrote:

Dear All thanks  so much for the suggestion !!!

One thing is not clear to me: How can write more safe string to send on
subprocess.Popen() without %s? What is the best way to do this?


The safest way is to use shell=False and pass a python list with the command 
line strings in it.


If you absolutely must generate a shell command string, you need to use some 
intermediate function that knows how to quote a string for the shell. Eg:


 def shell_quote(s):
   return "'" + s.replace("'", r"'\''") + "'"

That's untested, but it puts a string in single quotes and correctly escapes 
any single quotes in the string itself. Then you'd go:


 shcmd = "cat %s %s" % (shell_quote(filename1), shell_quote(filename2))
 P = Popen(shcmd, shell=True)

You will see the same kind of thing in most database interfaces, but presented 
more conveniently. As with the shell, it is always bad to go:


 sqlcmd = "INSERT into Table1 values(%s,%s)" % (value1, value2)

because value1 or value2 might have SQL punctuation in it. Eg:

 http://xkcd.com/327/

Instead you will usually use a call like this:

 db_handle.execute("INSERT into Table1 values(?,?)", value1, value2)

and the .execute function will itself call the right SQL quoting function and 
replace the "?" for you.


Cheers,
Cameron Simpson 

... It beeped and said "Countdown initiated." Is that bad?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess how to use?

[jarod...@libero.it]

Dear All thanks  so much for the suggestion !!!

One thing is not clear to me: How can write more safe string to send on 
subprocess.Popen() without %s? What is the best way to do this?

thanks so much!

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess how to use?

[Albert-Jan Roskam]

-
On Thu, Nov 6, 2014 12:57 AM CET Cameron Simpson wrote:

>On 05Nov2014 14:05, jarod...@libero.it  wrote:
>> I need to use external program from my scirpt.
>> code = "intersectBed -a %s -b /database/Refseq_Gene2.bed  -wa -wb|cut -f 
>> 4,8|uniq > tmp.tmp1"%("gene5.tmp.bed")
>>code2 = "intersectBed -a %s -b /database/Refseq_Gene2.bed -wa -wb|cut -f 
>> 4,8|uniq > tmp.tmp2"%("gene3.tmp.bed")
>>proc = []
>>p = subprocess.Popen(code,shell=True)
>>proc.append(p)
>>time.sleep(1) # seconds
>>p = subprocess.Popen(code2,shell=True)
>>proc.append(p)
>>time.sleep(1)
>>print 
>> >sys.stderr,'-'
>>print >sys.stderr,"Waiting for Star Fusion Annotate to finish running..."
>>for p in proc:
>>p.communicate()
>
>First remark: as written above, your shell pipelines do not read any input and 
>all their output goes to your temp files. Therefore using p.communicate() is a 
>waste of time. Just use p.wait(); there is no I/O to do from the point of view 
>of your Python code.
>
>Second remark: it is generally a bad idea to use Python's % operator to embed 
>strings in shell commands (or other "parsed by someone else" text, such as 
>SQL) because unless you are always very careful about the inserted string (eg 
>"gene5.tmp.bed") you may insert punctuation, leading to bad behaviour. In your 
>example code above you are sort of ok, and we can address this as a separate 
>issue after you have things working.
>
>>What append I don't habve any error however the file are truncated.
>
>You need to find out why.
>
>The shell code _will_ truncate "tmp.tmp1", but then we expect the output of 
>"uniq" to appear there.
>
>Start by stripping back the shell pipeline.
>
>If you remove the "|uniq", is there anything in "tmp.tmp1"?
>
>If there is not, strip off the "|cut -f 4,8" as well and check again. At that 
>point we're asking: does intersectBed actually emit any output?
>
>You can test that on the command line directly, and then build up the shell 
>pipeline to what you have above by hand. When working, _then_ put it into your 
>Python program.
>
>>So I try to use subprocess.call
>> 
>> p = subprocess.call(shlex.split(code),stdout = subprocess.PIPE, stderr = 
>> subprocess.STDOUT, shell = False)
>> but with p.comunicate I have this error:
>> 
>> ('\n* ERROR: Unrecognized parameter: -wb|cut *\n* ERROR: 
>> Unrecognized parameter: >
>[...]
>
>This is a standard mistake. shlex is  _not_ a full shell syntax parser. It is 
>a simple parser that understands basic shell string syntax (quotes etc) but 
>NOT redirections. It is really a tool for using with your own minilanguages, 
>as you might if you were writing an interactive program that accepts user 
>commands.
>
>You can't use shlex for shell pipelines or shell redirections.
>
>What is happening above is that all the "words" are being gathered up and 
>passed to the "intersectBed" command as arguments; no pipelines! And 
>"intersectBed" complains bitterly as you show.
>
>You can see this in the example you post below:
>
>> shlex.split(code)
>> Out[102]:
>> ['intersectBed',
>> '-a',
>> 'gene5.tmp.bed',
>> '-b',
>> '/home/maurizio/database/Refseq_Gene2.bed',
>> '-wa',
>> '-wb|cut',
>> '-f',
>> '4,8|uniq',
>> '>',
>> 'tmp.tmp1']
>> 
>> So what can I do? which system do you suggest to my Problem?
>
>First, forget about shlex. It does not do what you want.
>
>Then there are three approaches:
>
>1: Fix up your shell pipeline. You original code should essentially work: 
>there is just something wrong with your pipeline. Strip it back until you can 
>see the error.
>
>2: Construct the pipeline using Popen. This is complicated for someone new to 
>Popen and subprocess. Essentially you would make each pipeline as 3 distinct 
>Popen calls, using stdout=PIPE and attached that to the stdin of the next 
>Popen.

Look for "connecting segments" on 
http://pymotw.com/2/subprocess/


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess how to use?

[Cameron Simpson]

On 05Nov2014 14:05, jarod...@libero.it  wrote:

I need to use external program from my scirpt.
code = "intersectBed -a %s -b /database/Refseq_Gene2.bed  -wa -wb|cut -f 4,8|uniq > 
tmp.tmp1"%("gene5.tmp.bed")
   code2 = "intersectBed -a %s -b /database/Refseq_Gene2.bed -wa -wb|cut -f 4,8|uniq > 
tmp.tmp2"%("gene3.tmp.bed")
   proc = []
   p = subprocess.Popen(code,shell=True)
   proc.append(p)
   time.sleep(1) # seconds
   p = subprocess.Popen(code2,shell=True)
   proc.append(p)
   time.sleep(1)
   print 
>>sys.stderr,'-'
   print >>sys.stderr,"Waiting for Star Fusion Annotate to finish running..."
   for p in proc:
   p.communicate()


First remark: as written above, your shell pipelines do not read any input and 
all their output goes to your temp files. Therefore using p.communicate() is a 
waste of time. Just use p.wait(); there is no I/O to do from the point of view 
of your Python code.


Second remark: it is generally a bad idea to use Python's % operator to embed 
strings in shell commands (or other "parsed by someone else" text, such as SQL) 
because unless you are always very careful about the inserted string (eg 
"gene5.tmp.bed") you may insert punctuation, leading to bad behaviour. In your 
example code above you are sort of ok, and we can address this as a separate 
issue after you have things working.



   What append I don't habve any error however the file are truncated.


You need to find out why.

The shell code _will_ truncate "tmp.tmp1", but then we expect the output of 
"uniq" to appear there.


Start by stripping back the shell pipeline.

If you remove the "|uniq", is there anything in "tmp.tmp1"?

If there is not, strip off the "|cut -f 4,8" as well and check again. At that 
point we're asking: does intersectBed actually emit any output?


You can test that on the command line directly, and then build up the shell 
pipeline to what you have above by hand. When working, _then_ put it into your 
Python program.



   So I try to use subprocess.call

p = subprocess.call(shlex.split(code),stdout = subprocess.PIPE, stderr = 
subprocess.STDOUT, shell = False)
but with p.comunicate I have this error:

('\n* ERROR: Unrecognized parameter: -wb|cut *\n* ERROR: 
Unrecognized parameter: >

[...]

This is a standard mistake. shlex is  _not_ a full shell syntax parser. It is a 
simple parser that understands basic shell string syntax (quotes etc) but NOT 
redirections. It is really a tool for using with your own minilanguages, as you 
might if you were writing an interactive program that accepts user commands.


You can't use shlex for shell pipelines or shell redirections.

What is happening above is that all the "words" are being gathered up and 
passed to the "intersectBed" command as arguments; no pipelines! And 
"intersectBed" complains bitterly as you show.


You can see this in the example you post below:


shlex.split(code)
Out[102]:
['intersectBed',
'-a',
'gene5.tmp.bed',
'-b',
'/home/maurizio/database/Refseq_Gene2.bed',
'-wa',
'-wb|cut',
'-f',
'4,8|uniq',
'>',
'tmp.tmp1']

So what can I do? which system do you suggest to my Problem?


First, forget about shlex. It does not do what you want.

Then there are three approaches:

1: Fix up your shell pipeline. You original code should essentially work: there 
is just something wrong with your pipeline. Strip it back until you can see the 
error.


2: Construct the pipeline using Popen. This is complicated for someone new to 
Popen and subprocess. Essentially you would make each pipeline as 3 distinct 
Popen calls, using stdout=PIPE and attached that to the stdin of the next 
Popen.


3: The "cut" and "uniq" commands are very simple operations, and easy write 
directly in Python. Use Popen to dispatch _just_ "intersectBed" with 
stdout=PIPE. Then read from the pipe and do the cut and uniq yourself.


A completely untested and incomplete example of option 3 might look like this:

 P = Popen([ "intersectBed", ...other arguments... ], stdout=PIPE, shell-=False)
 old_fields = []
 for line in P.stdout:
   fields = line.split()   # read a line from "intersectBed" as words
   cut_fields = fields[3:8] # get columns 4..8 (counting from zero, not one)
   if cut_fields != old_fields:# this does what "uniq" does
 print cut_fields
 old_fields = cut_fields
 P.wait()  # finished reading output, wait for process and tidy 
up

Hopefully this suggests some ways forward.

Cheers,
Cameron Simpson 
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Subprocess how to use?

[jarod...@libero.it]

Dear All,


I need to use external program from my scirpt.
code = "intersectBed -a %s -b /database/Refseq_Gene2.bed  -wa -wb|cut -f 
4,8|uniq > tmp.tmp1"%("gene5.tmp.bed")
code2 = "intersectBed -a %s -b /database/Refseq_Gene2.bed -wa -wb|cut -f 
4,8|uniq > tmp.tmp2"%("gene3.tmp.bed")
proc = []
p = subprocess.Popen(code,shell=True)
proc.append(p)
time.sleep(1) # seconds
p = subprocess.Popen(code2,shell=True)
proc.append(p)
time.sleep(1)
print 
>>sys.stderr,'-'
print >>sys.stderr,"Waiting for Star Fusion Annotate to finish running..."
for p in proc:
p.communicate()
print 
>>sys.stderr,'-'


What append I don't habve any error however the file are truncated.
So I try to use subprocess.call 

 p = subprocess.call(shlex.split(code),stdout = subprocess.PIPE, stderr = 
subprocess.STDOUT, shell = False)
 but with p.comunicate I have this error:

 ('\n* ERROR: Unrecognized parameter: -wb|cut *\n* ERROR: 
Unrecognized parameter: > *\n* ERROR: Unrecognized parameter: tmp.tmp1 
*\n\nTool:bedtools intersect (aka intersectBed)\nVersion: 
v2.21.0-5-g479a2fc\nSummary: Report overlaps between two feature 
files.\n\nUsage:   bedtools intersect [OPTIONS] -a  -b 
\n\n\t\tNote: -b may be followed with multiple databases and/or 
\n\t\twildcard (*) character(s). \nOptions: \n\t-abam\tThe A input file is in 
BAM format.  Output will be BAM as well.\n\n\t-ubam\tWrite uncompressed BAM 
output. Default writes compressed BAM.\n\n\t-bed\tWhen using BAM input (-abam), 
write output as BED. The default\n\t\tis to write output in BAM when using 
-abam.\n\n\t-wa\tWrite the original entry in A for each 
overlap.\n\n\t-wb\tWrite the original entry in B for each overlap.\n\t\t- 
Useful for knowing _what_ A overlaps. Restricted by -f and 
-r.\n\n\t-loj\tPerform a "left outer join". That is, for each feature in 
A\n\t\treport each overlap with B.  If no overlaps are found, \n\t\treport a 
NULL feature for B.\n\n\t-wo\tWrite the original A and B entries plus the 
number of base\n\t\tpairs of overlap between the two features.\n\t\t- Overlaps 
restricted by -f and -r.\n\t\t  Only A features with overlap are 
reported.\n\n\t-wao\tWrite the original A and B entries plus the number of 
base\n\t\tpairs of overlap between the two features.\n\t\t- Overlapping 
features restricted by -f and -r.\n\t\t  However, A features w/o overlap are 
also reported\n\t\t  with a NULL B feature and overlap = 0.\n\n\t-u\tWrite the 
original A entry _once_ if _any_ overlaps found in B.\n\t\t- In other words, 
just report the fact >=1 hit was found.\n\t\t- Overlaps restricted by -f and 
-r.\n\n\t-c\tFor each entry in A, report the number of overlaps with B.\n\t\t- 
Reports 0 for A entries that have no overlap with B.\n\t\t- Overlaps restricted 
by -f and -r.\n\n\t-v\tOnly report those entries in A that have _no overlaps_ 
with B.\n\t\t- Similar to "grep -v" (an homage).\n\n\t-f\tMinimum overlap 
required as a fraction of A.\n\t\t- Default is 1E-9 (i.e., 1bp).\n\t\t- FLOAT 
(e.g. 0.50)\n\n\t-r\tRequire that the fraction overlap be reciprocal for A and 
B.\n\t\t- In other words, if -f is 0.90 and -r is used, this requires\n\t\t  
that B overlap 90% of A and A _also_ overlaps 90% of B.\n\n\t-s\tRequire same 
strandedness.  That is, only report hits in B\n\t\tthat overlap A on the _same_ 
strand.\n\t\t- By default, overlaps are reported without respect to 
strand.\n\n\t-S\tRequire different strandedness.  That is, only report hits in 
B\n\t\tthat overlap A on the _opposite_ strand.\n\t\t- By default, overlaps are 
reported without respect to strand.\n\n\t-split\tTreat "split" BAM or BED12 
entries as distinct BED intervals.\n\n\t-sorted\tUse the "chromsweep" algorithm 
for sorted (-k1,1 -k2,2n) input.\n\n\t-g\tProvide a genome file to enforce 
consistent chromosome sort order\n\t\tacross input files. Only applies when 
used with -sorted option.\n\n\t-header\tPrint the header from the A file prior 
to results.\n\n\t-nobuf\tDisable buffered output. Using this option will cause 
each line\n\t\tof output to be printed as it is generated, rather than 
saved\n\t\tin a buffer. This will make printing large output files 
\n\t\tnoticeably slower, but can be useful in conjunction with\n\t\tother 
software tools and scripts that need to process one\n\t\tline of bedtools 
output at a time.\n\n\t-names\tWhen using multiple databases, provide an alias 
for each that\n\t\twill appear instead of a fileId when also printing the DB 
record.\n\n\t-filenames\tWhen using multiple databases, show each complete 
filename\n\t\t\tinstead of a fileId when also printing the DB 
record.\n\n\t-sortout\tWhen using multiple databases, sort the output DB 
hits\n\t\t\tfor each record.\n\n\t-iobuf\tFollow with desired integer size of 
read buffer.\n\t\tOptional

Re: [Tutor] Subprocess help

[Danny Yoo]

> "Semantically, the command above means:
>   execute "./octosetupBROADCASTER-linux_i386.bin" in an environment
> that binds BYPASSROOT to "yes"."
>
> Does the above mean "BYPASSROOT" is an environmet variable? Is this created 
> and set by default by the system or is it created and set by 
> "octosetupBROADCASTER-linux_i386.bin?"


The word "system" is ambiguous.  Let's be specific.

The actor here is the command-line shell.

Assuming that you are using the 'bash' command-line shell, see:

http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_02.html

http://www.gnu.org/software/bash/manual/bash.html#Command-Execution-Environment
http://www.gnu.org/software/bash/manual/bash.html#Environment

for the background information.

The particular usage you are doing here at your command line shell:

$ BYPASSROOT=yes ./octosetupBROADCASTER-linux_i386.bin

is a concrete instance of augmenting the environment with a temporary
set of variables, as described in Section 3.7.4 of the Bash
documentation:

"""The environment for any simple command or function may be
augmented temporarily by prefixing it with parameter assignments, as
described in Shell Parameters. These assignment statements affect only
the environment seen by that command."""


I do need to cut in and editorialize: a little knowledge is dangerous.
You're asking questions that suggest that you have a passing
familiarity with Unix, but not a deep one.   Given what you appear to
be doing, this frightens me.

I am not joking; this is not a laughing matter.  Please talk to your
local friendly Unix system administrator, and take a real Unix system
administration class.  In the United States, you can visit one of the
USENIX groups:

https://www.usenix.org/legacy/membership/groups_usa.html

I'm not so familiar with the situation outside the US, but I'm pretty
sure similar groups can be found.



> The code you suggested...
>
> "envCopy = os.environ.copy()
>  envCopy['BYPASSROOT'] = 'yes'
>  subprocess.Popen(["./octosetupBROADCASTER-linux_i386.bin"], env=envCopy)"
>
> This copies the systems environment variables and stores them in the variable 
> "envCopy?" What do the "[ ]'s" mean? Are the variables stored in an array or 
> list and by enclosing "BYPASSROOT" in brackets, we are ultimately selecting 
> "BYPASSROOT" and assinging its value to "yes?"


Please read:

https://docs.python.org/2/tutorial/datastructures.html#dictionaries
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess help

[Crush]

Thank you Danny for your assistance. Just a couple of follow up questions to 
help me understand what is going on.

You said...

"Semantically, the command above means:
  execute "./octosetupBROADCASTER-linux_i386.bin" in an environment
that binds BYPASSROOT to "yes"."

Does the above mean "BYPASSROOT" is an environmet variable? Is this created and 
set by default by the system or is it created and set by 
"octosetupBROADCASTER-linux_i386.bin?"

The code you suggested...

"envCopy = os.environ.copy()
 envCopy['BYPASSROOT'] = 'yes'
 subprocess.Popen(["./octosetupBROADCASTER-linux_i386.bin"], env=envCopy)"

This copies the systems environment variables and stores them in the variable 
"envCopy?" What do the "[ ]'s" mean? Are the variables stored in an array or 
list and by enclosing "BYPASSROOT" in brackets, we are ultimately selecting 
"BYPASSROOT" and assinging its value to "yes?"

We then pass this to subprocess, but how does subprocess know we want 
"BYPASSROOT" to equal "yes" when we only pass on envCopy, which is coded before 
"
envCopy['BYPASSROOT'] = 'yes'?" Does python not work from top to bottom? 


Bo 

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess help

[Danny Yoo]

> The command I need to run is "BYPASSROOT=yes 
> ./octosetupBROADCASTER-linux_i386.bin"

Semantically, the command above means:

execute "./octosetupBROADCASTER-linux_i386.bin" in an environment
that binds BYPASSROOT to "yes".

The subprocess.Popen command takes in an optional "env" argument, so
that's what you want to provide when creating this new subprocess.
Here's a link to the documentation.

https://docs.python.org/2/library/subprocess.html#subprocess.Popen

So you'll probably want to do something like:

envCopy = os.environ.copy()
envCopy['BYPASSROOT'] = 'yes'
subprocess.Popen(["./octosetupBROADCASTER-linux_i386.bin"], env=envCopy)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Subprocess help

[Crush]

Hey all, hope everyone is well. 

I am trying to write a script that automates a certain task I have recently 
found myself doing a lot lately.

The command I need to run is "BYPASSROOT=yes 
./octosetupBROADCASTER-linux_i386.bin"

I know how to use subprocess to execute the "./octosetup..." command, however 
the "BYPASSROOT=yes" is stumping me seeing how it is not really a unix command. 
I assume it is changing a variable within "octosetup...?" How do I get the 
"BYPASSROOT=yes" to be included in the subprocess.call? 

I know i know "root" is bad, but with this, I have no choice. 

Thanks 

Bo 
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess not returning

[shawn wilson]

On Tue, Apr 22, 2014 at 3:10 PM, Alan Gauld  wrote:

> We have no clue what you are doing. You say "this works"
> but we can't see what 'this' is. Is the code on the
> pastebin link the working or the broken version?
>

Per what is expected output (which I forgot to provide - sorry about
that). Should be something like this:
[
  ['/testroot', '0', '0', '777'],
  ['/bin/dash', 0, 0, '755'],
  ['/lib/x86_64-linux-gnu/libc.so.6', '0', '0', '777'],
  ['/lib64/ld-linux-x86-64.so.2', '0', '0', '777'],
  ['/lib/x86_64-linux-gnu/libc-2.17.so', '0', '0', '755'],
  ['/lib/x86_64-linux-gnu/ld-2.17.so', '0', '0', '755']
]

Ie, find libraries a program is linked against (just try ldd against
any file because I'm not caring about optimizing at this point) and
then find the permissions of them and follow symlinks and do the same.

Though, what I'm asking specifically is why __ldd isn't returning any
values in my module.


The best I can simplify to show the part working that should also be
working in the gist code is:
import subprocess
import sys
import pprint
pp = pprint.PrettyPrinter(indent=4)

class T:
  def ldd(filename):
libs = []
for x in filename:
  p = subprocess.Popen(["ldd", x],
universal_newlines=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)

  for line in p.stdout:
s = line.split()
pp.pprint(s)
if "=>" in s:
  if len(s) == 3: # virtual library
continue
  else:
libs.append(s[2])
else:
  if len(s) == 2:
libs.append(s[0])

return libs

if __name__ == "__main__":
  t = T
  fattr = [
'/bin/dash'
  ]
  pp.pprint(["OUT", t.ldd(fattr)])


Past this, I can see that the ldd method is being called in my actual
code but nothing is being returned from it like it is here.

> It's also a very long listing. Can you produce a shorter
> example, perhaps with hard coded values that exhibits
> the problem?

I really did try to simplify t.py (where all of the data is included
in the script). I guess the reason for my question is because I'm not
sure what's not working or what to try next?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess not returning

[Alan Gauld]

On 22/04/14 10:14, shawn wilson wrote:

This works when I have a class for ldd and nothing else, but when I
run it like this:
https://gist.github.com/ag4ve/11171201

I don't get any of the libraries and I can't figure out where it's failing.
['fattr', [['/testroot', 0, 0, 777], ['/bin/dash']]]
HERE1 [/testroot]
HERE2 [/bin/dash]
['ldd', ['/lib/x86_64-linux-gnu/libc.so.6', '/lib64/ld-linux-x86-64.so.2']]
HERE2 [/lib/x86_64-linux-gnu/libc.so.6]
['ldd', ['/lib64/ld-linux-x86-64.so.2']]
HERE2 [/lib64/ld-linux-x86-64.so.2]
['ldd', ['statically']]
HERE1 [statically]
HERE2 [/lib64/ld-linux-x86-64.so.2]
['ldd', ['statically']]
HERE1 [statically]
[   'filelist',
 [['/testroot', 0, 0, 777], ['/bin/dash', 0, 0, '755'], [[[]], [

Obviously it's returning something - but no usable info.


We have no clue what you are doing. You say "this works"
but we can't see what 'this' is. Is the code on the
pastebin link the working or the broken version?

It's also a very long listing. Can you produce a shorter
example, perhaps with hard coded values that exhibits
the problem?

HTH
--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.flickr.com/photos/alangauldphotos

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess not returning

[shawn wilson]

This works when I have a class for ldd and nothing else, but when I
run it like this:
https://gist.github.com/ag4ve/11171201

I don't get any of the libraries and I can't figure out where it's failing.
['fattr', [['/testroot', 0, 0, 777], ['/bin/dash']]]
HERE1 [/testroot]
HERE2 [/bin/dash]
['ldd', ['/lib/x86_64-linux-gnu/libc.so.6', '/lib64/ld-linux-x86-64.so.2']]
HERE2 [/lib/x86_64-linux-gnu/libc.so.6]
['ldd', ['/lib64/ld-linux-x86-64.so.2']]
HERE2 [/lib64/ld-linux-x86-64.so.2]
['ldd', ['statically']]
HERE1 [statically]
HERE2 [/lib64/ld-linux-x86-64.so.2]
['ldd', ['statically']]
HERE1 [statically]
[   'filelist',
[['/testroot', 0, 0, 777], ['/bin/dash', 0, 0, '755'], [[[]], [

Obviously it's returning something - but no usable info.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess communications query

[William Ray Wing]

On Dec 10, 2013, at 2:28 PM, Reuben  wrote:

> Hi,
> 
> There exists two Linux machines A and B. Machine B contains python script 
> which needs to be run e.g. Test.py
> 
> In order to run that script, machine A needs to telnet into machine B and 
> then execute "python Test.py"
> 
> How can this be implemented? Is subprocess library to be used?if yes, an 
> example would help
> 
> Regards,
> Reuben
> 
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor

Check out pexpect here:  http://pexpect.sourceforge.net/pexpect.html

Should do exactly what you want, and Googling for pexpect examples will turn up 
lots of stuff.

-Bill___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess communications query

[Steven D'Aprano]

On Wed, Dec 11, 2013 at 12:58:16AM +0530, Reuben wrote:
> Hi,
> 
> There exists two Linux machines A and B. Machine B contains python script
> which needs to be run e.g. Test.py
> 
> In order to run that script, machine A needs to telnet into machine B and
> then execute "python Test.py"

Using telnet is not a good idea. Telnet is provably insecure -- it is a 
great big security hole. ssh is a better solution for machines which are 
on any untrusted network.


> How can this be implemented? Is subprocess library to be used?if yes, an
> example would help

An even better solution would be to use a library like Pyro or rpyc for 
executing remote procedure calls.

https://wiki.python.org/moin/DistributedProgramming

I strongly recommend you use an existing solution rather than hack 
together your own.



-- 
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess communications query

[Steve Willoughby]

Reuben  wrote:
>I want to implement a python script on machine A to do telnet/ssh into
>machine B (this might be easy)and then run the Test.py (this is
>challenging)
>On 11-Dec-2013 1:05 AM, "Danny Yoo"  wrote:
>
>> On Tue, Dec 10, 2013 at 11:28 AM, Reuben 
>wrote:
>> > Hi,
>> >
>> > There exists two Linux machines A and B. Machine B contains python
>script
>> > which needs to be run e.g. Test.py
>> >
>> > In order to run that script, machine A needs to telnet into machine
>B and
>> > then execute "python Test.py"
>>
>> Nothing about this sounds like Python.  Is there anything
>specifically
>> Python-related to this question, besides the detail that machine B is
>> running a Python program?
>>
>> This really sounds like more like a Linux system administration
>> question.  If that's the case, you probably want to ask on a forum
>for
>> system administrators, like:
>>
>> http://unix.stackexchange.com/
>>
>> where they'll be able to point you in a better direction than us.
>>
>> (By the way, when you say "telnet", I do hope you do not literally
>> mean telnet, which is not known to be secure.  You need to talk with
>> other system administrators and learn about tools like ssh.)
>>
>
>
>
>
>___
>Tutor maillist  -  Tutor@python.org
>To unsubscribe or change subscription options:
>https://mail.python.org/mailman/listinfo/tutor

You could do many different things depending on your needs. You could write a 
daemon in python on one machine which waits for a python program on the other 
to command it to run a process. 

You could find out how to accomplish this using ssh and then just write a 
python script to execute that ssh command for you. 

Without more details it's hard too say what is more appropriate but I would 
suggest reading up on python subprocess module, ssh, and internet protocol and 
server modules in the standard library.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess communications query

[Danny Yoo]

Ok; so in your situation, it sounds like machine A is also running a
Python script, and you want to automate the remote administration of
machine B through that program.  If that's the case, you may want to
look at the "Fabric" library, or other libraries that help with
driving ssh through Python:

http://docs.fabfile.org/en/1.8/

There are a few more links in:

https://wiki.python.org/moin/SecureShell

http://stackoverflow.com/questions/1233655/what-is-the-simplest-way-to-ssh-using-python
http://python-for-system-administrators.readthedocs.org/en/latest/index.html

that might be relevant to you.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess communications query

[Reuben]

I want to implement a python script on machine A to do telnet/ssh into
machine B (this might be easy)and then run the Test.py (this is challenging)
On 11-Dec-2013 1:05 AM, "Danny Yoo"  wrote:

> On Tue, Dec 10, 2013 at 11:28 AM, Reuben  wrote:
> > Hi,
> >
> > There exists two Linux machines A and B. Machine B contains python script
> > which needs to be run e.g. Test.py
> >
> > In order to run that script, machine A needs to telnet into machine B and
> > then execute "python Test.py"
>
> Nothing about this sounds like Python.  Is there anything specifically
> Python-related to this question, besides the detail that machine B is
> running a Python program?
>
> This really sounds like more like a Linux system administration
> question.  If that's the case, you probably want to ask on a forum for
> system administrators, like:
>
> http://unix.stackexchange.com/
>
> where they'll be able to point you in a better direction than us.
>
> (By the way, when you say "telnet", I do hope you do not literally
> mean telnet, which is not known to be secure.  You need to talk with
> other system administrators and learn about tools like ssh.)
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess communications query

[Danny Yoo]

On Tue, Dec 10, 2013 at 11:28 AM, Reuben  wrote:
> Hi,
>
> There exists two Linux machines A and B. Machine B contains python script
> which needs to be run e.g. Test.py
>
> In order to run that script, machine A needs to telnet into machine B and
> then execute "python Test.py"

Nothing about this sounds like Python.  Is there anything specifically
Python-related to this question, besides the detail that machine B is
running a Python program?

This really sounds like more like a Linux system administration
question.  If that's the case, you probably want to ask on a forum for
system administrators, like:

http://unix.stackexchange.com/

where they'll be able to point you in a better direction than us.

(By the way, when you say "telnet", I do hope you do not literally
mean telnet, which is not known to be secure.  You need to talk with
other system administrators and learn about tools like ssh.)
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Subprocess communications query

[Reuben]

Hi,

There exists two Linux machines A and B. Machine B contains python script
which needs to be run e.g. Test.py

In order to run that script, machine A needs to telnet into machine B and
then execute "python Test.py"

How can this be implemented? Is subprocess library to be used?if yes, an
example would help

Regards,
Reuben
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess module: when to _NOT_ use shell=True

[akleider]

Thank you "Eryksun" and "EikeWek" for your responses.  It is this sort of
thing that begins to pull one out of the newbie into the intermediate
category. I'm grateful.
alex

> I've not found anywhere a clear explanation of when not to set shell=True.
> If the command line must be interpreted by the shell then clearly this
> must be set. So the question that comes up is why not set it always?
> In an effort to investigate, I came up with the following script that
> indicates that the shell looks at only the first string in the array if
> the first parameter is an array rather than a string. Switching between
> cmd being a string vs an array and shell being set or not set gives 4
> possibilities.
> Any comments?
>
> #!/usr/bin/env python
>
> # file :  test.py (Python 2.7, NOT Python 3)
> # Running on Linux platform (Ubuntu.)
> print 'Running "tes.py"'
>
> import subprocess
>
> cmd = ["ls", "-l"]
> # cmd = "ls -l"
> p = subprocess.Popen(cmd,
> # shell=True,
> stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> (s_out, s_err) = p.communicate()
> print "Std_out returns:\n%s\nStd_err returns:\n%s\n"%\
>   (s_out, s_err, )
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
>


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess module: when to _NOT_ use shell=True

[Eike Welk]

On Sunday 10.03.2013 09:56:26 aklei...@sonic.net wrote:
> I've not found anywhere a clear explanation of when not to set shell=True.
> If the command line must be interpreted by the shell then clearly this
> must be set. So the question that comes up is why not set it always?

Because ``shell=True`` is a security problem. It it is also not portable. 
Someone might want to run your code on windows, which has no Bash.

The security problem arises when the command contains any user input. The user 
could enter a bit of carefully crafted text, that tricks Bash into doing 
something that you don't want. The technique is called "shell code injection".

The nicest example is the "Bobby tables" episode from XKCD, that covers a 
similar situation with SQL injection:

http://www.explainxkcd.com/wiki/index.php?title=327:_Exploits_of_a_Mom

And on Wikipedia:

http://en.wikipedia.org/wiki/Code_injection#Shell_injection


--
Eike.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess module: when to _NOT_ use shell=True

[eryksun]

On Sun, Mar 10, 2013 at 12:56 PM,   wrote:
> I've not found anywhere a clear explanation of when not to
> set shell=True. If the command line must be interpreted by
> the shell then clearly this must be set. So the question
> that comes up is why not set it always?

Using the shell can be a security risk for untrusted commands, as
described in the 3.3 docs for shlex.quote:

http://docs.python.org/3/library/shlex#shlex.quote

> I came up with the following script that indicates that
> the shell looks at only the first string in the array if
> the first parameter is an array rather than a string.
> Switching between cmd being a string vs an array and shell
> being set or not set gives 4 possibilities.
>
> cmd = ["ls", "-l"]
> # cmd = "ls -l"

On a POSIX system, when you use an ags string instead of a list with
Popen, it just adds the string to a list, which varies depending on
the "shell" argument.

Starting a new process using fork/exec hasn't fundamentally changed
since the early Unix systems in the 1970s. A child process is forked
and executes a new process image, with the given arguments in an array
of pointers to strings. Popen uses uses os.fork and os.execvp (or
os.execvpe if you supply an environment).

http://docs.python.org/2/library/os#os.fork
http://docs.python.org/2/library/os#os.execvp

http://en.wikipedia.org/wiki/Exec_%28operating_system%29

If shell=False, use the args list ["ls", "-l"]. Otherwise, if you use
an args string, Popen creates the list ["ls -l"], and execvp will look
for a file named "ls -l". Here's a silly example:

>>> import os
>>> from subprocess import Popen

>>> os.environ['PATH'] += ':.'
>>> open('ls -l', 'w').write('''\
... #!/bin/bash
... echo silliness''')
>>> os.chmod('ls -l', 0700)
>>> p = Popen('ls -l')
>>> silliness

If shell=True and the command is the string "ls -l", Popen uses the
args list ["/bin/sh", "-c", "ls -l"]. This is equivalent to running
the following:

/bin/sh -c 'ls -l'

This will work as expected. If you instead use the list ["ls", "-l"],
Popen uses the args list ["/bin/sh", "-c", "ls", "-l"], which is
equivalent to running the following:

/bin/sh -c ls -l

You can verify that the above doesn't work (the '-l' option isn't
passed to ls). Here's an example to echo the parameters:

>>> open('tmp.sh', 'w').write('''
... #!/bin/bash
... echo $0, $1, $2''')
>>> os.chmod('tmp.sh', 0700)
>>> env = {'PATH':'.'}

>>> p = Popen('tmp.sh p1 p2', shell=True, env=env)
>>> ./tmp.sh, p1, p2

That worked fine, but this fails:

>>> p = Popen(['tmp.sh','p1','p2'], shell=True, env=env)
>>> ./tmp.sh, ,
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess module: when to _NOT_ use shell=True

[akleider]

I've not found anywhere a clear explanation of when not to set shell=True.
If the command line must be interpreted by the shell then clearly this
must be set. So the question that comes up is why not set it always?
In an effort to investigate, I came up with the following script that
indicates that the shell looks at only the first string in the array if
the first parameter is an array rather than a string. Switching between
cmd being a string vs an array and shell being set or not set gives 4
possibilities.
Any comments?

#!/usr/bin/env python

# file :  test.py (Python 2.7, NOT Python 3)
# Running on Linux platform (Ubuntu.)
print 'Running "tes.py"'

import subprocess

cmd = ["ls", "-l"]
# cmd = "ls -l"
p = subprocess.Popen(cmd,
# shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(s_out, s_err) = p.communicate()
print "Std_out returns:\n%s\nStd_err returns:\n%s\n"%\
  (s_out, s_err, )


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess Popen PIPE error

[Abhijeet Rastogi]

Ya. Do it using python. Why do you want to use bash when you already have
python?

See, do something like this:-

import os
alldirs = os.listdir("/path/to/dir")

DIRS = [] #Only the dirs you are interested in

for i in alldirs:
  if i.find("deploy") is -1: L.append(i)
  if i.find("TEMPLATE") is -1: L.append(i)

#Now L contains all the required dirs.

Hope it helps.

On Tue, Oct 26, 2010 at 9:19 PM, Sean Carolan  wrote:

> > Here is the bash one-liner that generates my list,
> > with one name per line:
> >
> > ls -d */ | grep -v -E 'deploy|TEMPLATE' | sed 's/\///'
> >
> > How would you get the output of this into a python list that could
> > then be used in the script?  Please forgive my ignorance; I've read
> > through the documentation but am still not clear on this aspect.
>
> Would this be easier to accomplish using os.listdir()?.  Basically I
> want to make a list of directories, excluding files and the
> directories or files containing "deploy" and "TEMPLATE".
>



-- 
Abhijeet Rastogi (shadyabhi)
http://www.google.com/profiles/abhijeet.1989
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess Popen PIPE error

[Abhijeet Rastogi]

Please read the documentation of Popen. You cannot pass arguments like that.

>>> from subprocess import Popen,PIPE
>>> import shlex
>>> cmd="ls -l"
>>> cmd=shlex.split(cmd)
>>> p = Popen(cmd,stdout=PIPE,stderr=PIPE)

or simply that means

>>> p = Popen(['ls','-l'],stdout=PIPE,stderr=PIPE)

Hope I have made my point clear. The problem occuring is that there is no
binary that has name as "ls -l". Python cannot understand that "-l" is an
argument until you pass it this way.


On Tue, Oct 26, 2010 at 6:52 PM, Sean Carolan  wrote:

> What am I doing wrong here?
>
> >>> from subprocess import Popen, PIPE
> >>> cmd = 'ls -l'
> >>> p = Popen(cmd, stdout=PIPE, stderr=PIPE)
> Traceback (most recent call last):
>  File "", line 1, in ?
>  File "/usr/lib64/python2.4/subprocess.py", line 550, in __init__
>errread, errwrite)
>  File "/usr/lib64/python2.4/subprocess.py", line 993, in _execute_child
>raise child_exception
> OSError: [Errno 2] No such file or directory
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>



-- 
Abhijeet Rastogi (shadyabhi)
http://www.google.com/profiles/abhijeet.1989
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess Popen PIPE error

[Sean Carolan]

What am I doing wrong here?

>>> from subprocess import Popen, PIPE
>>> cmd = 'ls -l'
>>> p = Popen(cmd, stdout=PIPE, stderr=PIPE)
Traceback (most recent call last):
  File "", line 1, in ?
  File "/usr/lib64/python2.4/subprocess.py", line 550, in __init__
errread, errwrite)
  File "/usr/lib64/python2.4/subprocess.py", line 993, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess output

[Evert Rol]

> #!/usr/bin/env python
> import subprocess
> import sys
> 
> dom = sys.argv[1]
> switch = sys.argv [2]
> answer = subprocess.call("whois " + dom, shell=True)
> 
> Execute the above (whatever.py -example.com -a1) prints the entire WHOIS
> output, but I just want it saved to the variable 'answer', nothing more.

create a subprocess.Popen object instead, and use communicate().
Use subprocess.PIPE for the stdout and stderr arguments (and use a list for the 
executable + arguments: ["whois", dom], leaving out the shell argument).
Read the library docs: see example 17.1.3.2.


> Changing 'shell=True' to 'shell=False' raises an exception and removing
> the 'shell=' altogether is troublesome as well.

shell=False is the default

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess output

[Sander Sweers]

- Original message -
> I'm using Python 2.6.5 and I've got a challenge with the subprocess
> module. I'd like the output to be stored in a variable, and not sent to
> the stdout. The relevant lines as they are now:

Go to google, enter "pymotw subprocess" and hit I am feeling lucky ;-)

greets
Sander

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess output

[ashley]

I'm using Python 2.6.5 and I've got a challenge with the subprocess
module. I'd like the output to be stored in a variable, and not sent to
the stdout. The relevant lines as they are now:

#!/usr/bin/env python
import subprocess
import sys

dom = sys.argv[1]
switch = sys.argv [2]
answer = subprocess.call("whois " + dom, shell=True)

Execute the above (whatever.py -example.com -a1) prints the entire WHOIS
output, but I just want it saved to the variable 'answer', nothing more.

Changing 'shell=True' to 'shell=False' raises an exception and removing
the 'shell=' altogether is troublesome as well.

Thoughts on how I can accomplish this?

Thanks!


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess

[Sander Sweers]

On 27 March 2010 09:30, David Abbott  wrote:
> Here is an example using subprocess.call
> http://dwabbott.com/code/index8.html
>
> and some more here with subprocess.Popen
> http://asterisklinks.com/wiki/doku.php?id=wiki:subprocess

On top of that we have the excelent PyMOTW from Doug on subprocess.
http://blog.doughellmann.com/2007/07/pymotw-subprocess.html

Greets
Sander
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess

[kevin parks]

Thanks David. Those are excellent short clear examples. I will look those over. 
Super! Thanks for that.

-kp

On Mar 27, 2010, at 5:30 PM, David Abbott wrote:

> On Sat, 2010-03-27 at 16:55 +0900, kevin parks wrote:
> 

> Here is an example using subprocess.call
> http://dwabbott.com/code/index8.html
> 
> and some more here with subprocess.Popen
> http://asterisklinks.com/wiki/doku.php?id=wiki:subprocess
> 
> HTH
> David
> 
> 

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess

[David Abbott]

On Sat, 2010-03-27 at 16:55 +0900, kevin parks wrote:
> I tried readings some toots and tried reading alan's thing. I just still 
> can't grok how to use subprocess.
> 
> I am trying to call sox (fun fact: an early contributer to sox was none other 
> than Guido van Rossum)
> 
> In the old days you would just use os i guess, like:
> 
> import os
> os.system('sox -V3 -D -S St.01.aif -b16 Stout-01.aif rate -s -v 44100')
> 
> to call a unix executable that you would ordinarily run from the terminal.
> 
> what would the equivalent of this in python's new subprocess be? perhaps if i 
> saw an example it would click..
> 
> additionally.. sox is a sound conversion tool. I plan to batch process a 
> bunch of files. Will subprocess start all the jobs as it finds the files? or 
> will it process one job and que the next? If it opened a thread and started a 
> bunch of jobs it would likely bog down the system no?
> 
> anyway  I have the os walk and other stuff that i am working on and i was 
> hoping to get some help on subprocess. There are a few pages on subprocess 
> but they all might just as well be in chinese and none of them, none that i 
> can see are equivalent to what i am trying to do (they all seem to be doing 
> os-y things)
> 
> An example might help. Not sure why i am finding this so hard to get my head 
> around.

Here is an example using subprocess.call
http://dwabbott.com/code/index8.html

and some more here with subprocess.Popen
http://asterisklinks.com/wiki/doku.php?id=wiki:subprocess

HTH
David


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess

[kevin parks]

I tried readings some toots and tried reading alan's thing. I just still can't 
grok how to use subprocess.

I am trying to call sox (fun fact: an early contributer to sox was none other 
than Guido van Rossum)

In the old days you would just use os i guess, like:

import os
os.system('sox -V3 -D -S St.01.aif -b16 Stout-01.aif rate -s -v 44100')

to call a unix executable that you would ordinarily run from the terminal.

what would the equivalent of this in python's new subprocess be? perhaps if i 
saw an example it would click..

additionally.. sox is a sound conversion tool. I plan to batch process a bunch 
of files. Will subprocess start all the jobs as it finds the files? or will it 
process one job and que the next? If it opened a thread and started a bunch of 
jobs it would likely bog down the system no?

anyway  I have the os walk and other stuff that i am working on and i was 
hoping to get some help on subprocess. There are a few pages on subprocess but 
they all might just as well be in chinese and none of them, none that i can see 
are equivalent to what i am trying to do (they all seem to be doing os-y things)

An example might help. Not sure why i am finding this so hard to get my head 
around.






___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Subprocess Stdout Redirection

[Gil Johnson]

Muhammad,
I am no expert, but I think you have to call proc_log.close() in your
StartProc() to make sure that the file is written out to disk.
If I understand what you are doing, you are trying to generate a new path for
each pass through the loop, of the form 0/, 1/,... and write "MyLog.txt" to the
new directory.
When I did something like this in the past I put the files MyLog0.txt,
MyLog1.txt, MyLog2.txt,... in a single directory, which was simpler and more
convenient for me when I gathered the information together for analysis:

def NewProc(FileNum):
 filename = "MyLog" + repr(FileNum) + ".txt"
 proc_log = open(filename,"w")
 proc_log.write("Output here")
 proc_log.close()
 return

for i in range(0, 5):
NewProc(i)


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Subprocess Stdout Redirection

[Muhammad Ammar]

Hi,

Hope you will be fine and good.


Following is a snippet of python code i am using for a regression testing.

def StartProc(dir, parm):

global proc

proc_log = open(dir + os.sep + "MyLog.txt","w")  #new path for each file


if parm:
proc = subprocess.Popen(path, 0, None, subprocess.PIPE, proc_log, None)

else:
MyReset(proc) #reset the process(proc) to
its default values

proc.stdout = proc_log#no effect
print "fptr ", proc.stdout

#endif
#enddef

prm = True


for i in range(0, 5):

StartProc(i, prm)
prm = False

#endfor

What I want to do is to start an executable only once, but on each iteration
I want to redirect the process output to a different file. What is
happening, is that files are created in the different path, but output is
redirected to the file that is created first time.

Note: MyReset() initializes the process (executable) to its default values
after the first iteration.

Will the following line change the process stdout to new file?
proc.stdout = proc_log

If no, than can you kindly guide me to the right way

*
Note: In the StartProc() if i create a new process everytime
**StartProc**() is called. Output is correctly redirected to proper file.*


Waiting for you kind reply.

Warm Regards,
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess/files help

[Lie Ryan]

Essah Mitges wrote:
> game runs fine out of the WODDS.py it when I start it from the menu lies the 
> problemup-1.png is the first pic file used in the program so I'm guessing 
> that all of the imported files from data have a problem if i took away up-1 
> up-2 would start a problem then side-1 and so on
> 

Let me guess, the menu script is in different directory than the game
script? The current working directory is thus pointing to the menu's
directory instead of script's.

Two possible fix is to 1) move the menu script to the game's script
directory or 2) pass cwd="" argument to subprocess.Popen

However, as others have said, both of them are the WRONG approach to
your problem... since both scripts are python scripts, both under your
control, and most importantly both scripts are integrated part of a
unity (i.e. the game); it will be MUCH easier to import the game script
and call the function that started the game.

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess/files help

[Essah Mitges]

game runs fine out of the WODDS.py it when I start it from the menu lies the 
problemup-1.png is the first pic file used in the program so I'm guessing that 
all of the imported files from data have a problem if i took away up-1 up-2 
would start a problem then side-1 and so on


> From: sri...@gmail.com
> Date: Wed, 10 Jun 2009 17:16:43 -0500
> Subject: Re: [Tutor] subprocess/files help
> To: e_mit...@hotmail.com
> CC: tutor@python.org
>
> On Wed, Jun 10, 2009 at 4:21 PM, Essah Mitges> wrote:
>
>
>
>
> The problem I am have now is that once my game is initiated from the menu 
> file the modules that are with cannot import the imagesthe folder looks like 
> thisWODDS data All images and sound for game gamelib Objects Utilities 
> ___init___ WODDS.py
>
>
>
> Png error file attachedAny one know the code in subprocesses to fix this
>
> The problem has nothing to do with subprocess (and in the future please 
> combine your questions into one email). As I mentioned before it's 
> specifically looking for a file "data/up-1.png" which either doesn't exist or 
> permissions are wrong.
>
>
>
> Remember, relative paths are not absolute.
>
> On *nix, /home/user/file/image.jpg is an absolute reference. No matter where 
> you execute a program it will be able to find it. On Windows you would see 
> something like C:\Documents and Settings\Users\MyGuy\MyFile\image.jpg
>
>
>
> A relative path would be, as the name suggests, relative. files/image.jpg 
> will access a different location on the disk depending on where you are. Take 
> a look at this that specifically deals with web programming but is applicable.
>
>
> http://www.communitymx.com/content/article.cfm?cid=230ad
>
> HTH
>
>

_
Internet explorer 8 lets you browse the web faster.
http://go.microsoft.com/?linkid=9655582
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess/files help

[Wayne]

On Wed, Jun 10, 2009 at 4:21 PM, Essah Mitges  wrote:

>
> The problem I am have now is that once my game is initiated from the menu
> file the modules that are with cannot import the imagesthe folder looks like
> thisWODDSdataAll images and sound for gamegamelib
>  ObjectsUtilities___init___WODDS.py
> Png error file attachedAny one know the code in subprocesses to fix this


The problem has nothing to do with subprocess (and in the future please
combine your questions into one email). As I mentioned before it's
specifically looking for a file "data/up-1.png" which either doesn't exist
or permissions are wrong.

Remember, relative paths are not absolute.

On *nix, /home/user/file/image.jpg is an absolute reference. No matter where
you execute a program it will be able to find it. On Windows you would see
something like C:\Documents and Settings\Users\MyGuy\MyFile\image.jpg

A relative path would be, as the name suggests, relative. files/image.jpg
will access a different location on the disk depending on where you are.
Take a look at this that specifically deals with web programming but is
applicable.
http://www.communitymx.com/content/article.cfm?cid=230ad

HTH
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess Popen

[Tiago Saboga]

On Fri, Apr 03, 2009 at 03:12:08PM -0700, Weidner, Ronald wrote:
> I have a long running py script that I'm trying to kick off from
> another long running py script as a separate process... If either
> script fails or terminates, I don't want the other script to be
> effected as a result. In other words I need a separate process not a
> new thread. In any case, this code I thought would work but it
> doesn't.
> 

> someLongRunningScript= ( "%s/someLongRunningScript.py" % ( os.getcwd() ) )
> someLongRunningScriptArgs= ( '--verbose --directory=%s --link=%s %s' % ( 
> directory_name, link_name, span_option ) )
> longCommand = ( '%s %s' % ( someLongRunningScript, someLongRunningScriptArgs) 
> )
> pid = subprocess.Popen ( ["/usr/bin/python", longCommand ] ).pid
> print ( "/usr/bin/python %s " % ( longCommand ) )

> 
> What's interesting is that if I copy the text printed in the 5th
> line, paste into my shell, then run it -- the process I'm trying to
> start works perfectly.
> 
> The problem seems to have something to do with the way arguments are
> being passed to the python script named someLongRunningProcess.py.

Yes, the arguments are not passed to the shell. There is a Popen
option to pass them to a shell, perhaps it would work. But the simpler
way is to write:

pid = subprocess.Popen(["/usr/bin/python"] + longCommand.split()).pid

It will not work if some arguments are quoted strings with spaces. It
would be better if longCommand was already a list.

Tiago.
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess Popen

[Weidner, Ronald]

I have a long running py script that I'm trying to kick off from another long 
running py script as a separate process... If either script fails or 
terminates, I don't want the other script to be effected as a result. In other 
words I need a separate process not a new thread. In any case, this code I 
thought would work but it doesn't.

someLongRunningScript= ( "%s/someLongRunningScript.py" % ( os.getcwd() ) )
someLongRunningScriptArgs= ( '--verbose --directory=%s --link=%s %s' % ( 
directory_name, link_name, span_option ) )
longCommand = ( '%s %s' % ( someLongRunningScript, someLongRunningScriptArgs) )
pid = subprocess.Popen ( ["/usr/bin/python", longCommand ] ).pid
print ( "/usr/bin/python %s " % ( longCommand ) )

What's interesting is that if I copy the text printed in the 5th line, paste 
into my shell, then run it -- the process I'm trying to start works perfectly.

The problem seems to have something to do with the way arguments are being 
passed to the python script named someLongRunningProcess.py.

Help is greatly appreciated.

--
Ron
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess Popen

[Weidner, Ronald]

I have a long running py script that I'm trying to kick off from another long 
running py script as a separate process... If either script fails or 
terminates, I don't want the other script to be effected as a result. In other 
words I need a separate process not a new thread. In any case, this code I 
thought would work but it doesn't.

someLongRunningScript= ( "%s/someLongRunningScript.py" % ( os.getcwd() ) )
someLongRunningScriptArgs= ( '--verbose --directory=%s --link=%s %s' % ( 
directory_name, link_name, span_option ) )
longCommand = ( '%s %s' % ( someLongRunningScript, someLongRunningScriptArgs) )
pid = subprocess.Popen ( ["/usr/bin/python", longCommand ] ).pid
print ( "/usr/bin/python %s " % ( longCommand ) )

What's interesting is that if I copy the text printed in the 5th line, paste 
into my shell, then run it -- the process I'm trying to start works perfectly.

The problem seems to have something to do with the way arguments are being 
passed to the python script named someLongRunningProcess.py.

Help is greatly appreciated.

--
Ron

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess adds %0A to end of string

[Martin Walsh]

David wrote:
> Martin Walsh wrote:
> 
>> Welcome!
> 
> thanks

welcome (uh oh, infinite loop warning)

> This "podcast_file.write('%s: %s' % (entry.updated, entry.link))"
> writes it in one very long string

Copy and paste gets me every time. Try this, and note the presence of
the newline ('\n'):

podcast_file.write('%s: %s\n' % (entry.updated, entry.link))

> 
> The Latest Link
> http://linuxcrazy.com/podcasts/LC-44-arne.ogghttp://linuxcrazy.com/podcas=>>
> 
> 
> and sys.stdout prints to the file one line at a time

The primary issue is not that you're writing to sys.stdout, it's that
you're using the print statement which implicitly adds a newline.
Technically you can also do the following (python <= 2.5, and maybe 2.6)
which should demonstrate the concept...

podcastfile = file(somepath, 'a')
print >> podcastfile, '%s: %s' (entry.updated, entry.link)

... however, I would recommend against using the above print statement
syntax for anything other than experimentation because 1) it's not
common practice IMHO, and 2) it's gone starting with python 3.0. 'print'
will be a builtin function instead of a statement going forward.

I second Alan's recommendation to read a tutorial or two to solidify
your understanding of the basic concepts. Alan's tutorial is very good,
and covers everything we have discussed so far, particularly the
"Handling Files" section. Check the link in his email sig.

HTH,
Marty
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess adds %0A to end of string

[Alan Gauld]

"David"  wrote


sys.stdout = open("podcast_links.txt", "a")
print '%s' % (entry.link)
sys.stdout.close()
getFeed()



This "podcast_file.write('%s: %s' % (entry.updated, entry.link))"
writes it in one very long string


Use podcastfile.writeline() to write it line by line.

Read any tutorial on file handling to understand how/why these 
various methods work. Get to know files, you'll use them a lot!



--
Alan Gauld
Author of the Learn to Program web site
http://www.freenetpages.co.uk/hp/alan.gauld


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess adds %0A to end of string

[David]

Martin Walsh wrote:



Welcome!



thanks


You should probably try to avoid reassigning sys.stdout. This is usually
a bad idea, and can cause odd behavior that is difficult to
troubleshoot, especially for a beginner. A reasonable approach is to
assign the open file object to a name of your own choosing...

.>>> podcast_file = open('podcast.txt', 'a')

... and then, use the write method of the file object ...

.>>> podcast_file.write('%s: %s' % (entry.updated, entry.link))

More info here:
http://www.python.org/doc/2.5.3/tut/node9.html#SECTION00920



print '%s: %s' % (entry.updated, entry.link)
sys.stdout.close()
for entry in data.entries:
sys.stdout = open("podcast_links.txt", "a")
print '%s' % (entry.link)
sys.stdout.close()
getFeed()

This "podcast_file.write('%s: %s' % (entry.updated, entry.link))"
writes it in one very long string

The Latest Link
http://linuxcrazy.com/podcasts/LC-44-arne.ogghttp://linuxcrazy.com/podcas=>>

and sys.stdout prints to the file one line at a time

How do I split the long string, not even sure if that is the correct term?

for i in somefile: ?

re.somefile(do_something) ?



--
Powered by Gentoo GNU/LINUX
http://www.linuxcrazy.com
pgp.mit.edu

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess adds %0A to end of string

[Martin Walsh]

Hi David,

David wrote:
> Hi everyone.
> Just learning :) I have one program to parse a podcast feed and put it
> into a file.

Welcome!


> 
> def getFeed():
> url = raw_input("Please enter the feed: ")
> data = feedparser.parse(url)
> for entry in data.entries:
> sys.stdout = open("podcast.txt", "a")

You should probably try to avoid reassigning sys.stdout. This is usually
a bad idea, and can cause odd behavior that is difficult to
troubleshoot, especially for a beginner. A reasonable approach is to
assign the open file object to a name of your own choosing...

.>>> podcast_file = open('podcast.txt', 'a')

... and then, use the write method of the file object ...

.>>> podcast_file.write('%s: %s' % (entry.updated, entry.link))

More info here:
http://www.python.org/doc/2.5.3/tut/node9.html#SECTION00920


> print '%s: %s' % (entry.updated, entry.link)
> sys.stdout.close()
> for entry in data.entries:
> sys.stdout = open("podcast_links.txt", "a")
> print '%s' % (entry.link)
> sys.stdout.close()
> getFeed()
> 
> next to get the latest item;
> 

> lname = "podcast_links.txt"
> L = open(lname, 'r')
> print "The Latest Link\n"
> download = L.readline()

The readline method returns a line from the file *including* the newline
 character(s) ('\n').

> print download
> 
> answer = raw_input("Do you want to download the podcast? ")
> if answer == "y":
> wget = "wget"
> subprocess.call([wget, download])
> else:
> print "oops"

OK. There's the problem. Let's assume that after 'download =
L.readline()' that download equals this (you can confirm by adding a
'print repr(download)'):

'http://linuxcrazy.com/podcasts/LC-44-arne.mp3\n'

... then the call becomes (behind the scenes)

subprocess.call(['wget',
'http://linuxcrazy.com/podcasts/LC-44-arne.mp3\n'])

... so the newline is passed as part of the first argument to the wget
command.

Not so coincidentally, the '%0A' represents a newline ('\n') in a
properly quoted/escaped URL.

.>>> import urllib2
.>>> urllib2.unquote('%0A')
'\n'

I suspect it is the wget command which is quoting the newline, not the
subprocess call, as subprocess doesn't know anything about valid
characters for urls.

You can work around this problem as you already have by dropping the
last character as in 'download[:-1]', or use the strip (or rstrip) str
method:

.>>> download.rstrip()
'http://linuxcrazy.com/podcasts/LC-44-arne.mp3'

More info here: http://www.python.org/doc/2.5.3/lib/string-methods.html

HTH,
Marty
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess adds %0A to end of string

[Kent Johnson]

On Sun, Dec 21, 2008 at 7:29 PM, David  wrote:
> This seems to work;
>
> download = L.readline()
> print download
> download = download[0:-1]
>
> What is that last character that is added;
> .mp3%0A

It is a newline character (line feed). readline() includes the line
endings in the returned lines. Try
download = L.readline().rstrip()

Kent
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess adds %0A to end of string

[David]

This seems to work;

download = L.readline()
print download
download = download[0:-1]

What is that last character that is added;
.mp3%0A


--
Powered by Gentoo GNU/LINUX
http://www.linuxcrazy.com
pgp.mit.edu

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess adds %0A to end of string

[David]

Hi everyone.
Just learning :) I have one program to parse a podcast feed and put it 
into a file.


#!/usr/bin/python
"""Get feed date and link details"""
import feedparser
import sys

def getFeed():
url = raw_input("Please enter the feed: ")
data = feedparser.parse(url)
for entry in data.entries:
sys.stdout = open("podcast.txt", "a")
print '%s: %s' % (entry.updated, entry.link)
sys.stdout.close()
for entry in data.entries:
sys.stdout = open("podcast_links.txt", "a")
print '%s' % (entry.link)
sys.stdout.close()
getFeed()

next to get the latest item;

#!/usr/bin/python
"""read the last entry and download podcast"""
import subprocess
fname = "podcast.txt"
f = open(fname, 'r')
print "The Latest Podcast", f.readline()
print f.readline()
f.close()
lname = "podcast_links.txt"
L = open(lname, 'r')
print "The Latest Link\n"
download = L.readline()
print download

answer = raw_input("Do you want to download the podcast? ")
if answer == "y":
wget = "wget"
subprocess.call([wget, download])
else:
print "oops"

and here is the output;

david [06:37 PM] opteron ~ $ ./py_read_podcast.py
The Latest Podcast

Sat, 20 Dec 2008 01:52:00 GMT: http://linuxcrazy.com/podcasts/LC-44-arne.mp3

The Latest Link

http://linuxcrazy.com/podcasts/LC-44-arne.mp3

Do you want to download the podcast? y
--2008-12-21 18:38:46--  http://linuxcrazy.com/podcasts/LC-44-arne.mp3%0A
Resolving linuxcrazy.com... 74.220.207.171
Connecting to linuxcrazy.com|74.220.207.171|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2008-12-21 18:38:46 ERROR 404: Not Found.

See how it adds the %0A to the end ?

I am just trying to get it to work, I know it needs a lot of work, error 
checking etc. And I am sure there are better ways, I am new.


thanks
-david


--
Powered by Gentoo GNU/LINUX
http://www.linuxcrazy.com
pgp.mit.edu

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Subprocess module

[Judith Flores]

Hello,

   A couple of weeks ago I posted a question about what documentation I should 
read in order to implement a way to communicate Python and R. I read about the 
module 'subprocess', but have not been able to do something very simple. I was 
wondering if you could tell me how to transfer a Python variable to R. For 
example, a variable called 'fileforR'  will acquire a pathfile from the user 
through a GUI. This file will be a csv file that will be processed in a R 
script for statistical analysis. Let's suppose the R script is called: 
'read_file.R', and that it contains the following code:

read.csv(fileforR)
q()

In the Python interpreter:

import easygui
import subprocess

fileforR = fileopenbox()

After this, I have tried different combinations to try to call R and the 
mentioned R script, for example:

subprocess.call('R')

opens the R console, but I can't access the R script. And there's another 
problem I can't solve: passing the 'fileforR' variable from Python to the R 
environment.


   I am working on SUSE Linux, and using Python 2.5.2.


Thank you very much,

Judith


  
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess help, nohup

[Martin Walsh]

John wrote:
> Hello,

Hi John,

I didn't see a response to your question, so I'll make an attempt ...

>  
> I've written a script which conducts several subprocess calls and then
> ultimately calls a shell script which runs even more programs... my
> script is using subprocess to execute a few sed calls, and then execute
> the script. I'm getting strange behavior:

Just a suggestion, if you find you are preforming sed replace operations
regularly, you might consider writing your own sed-like replace
function. This way you avoid spawning a subprocess altogether for
search/replace functionality, and as an added bonus you can re-use it in
future scripts. The following (untested) code should get you started, if
you choose to go this route:

import re

def sed_replace(search, replace, text):
pattern = re.compile(search)
return pattern.sub(replace, text)

def infile_sed_replace(search, replace, path):
text = file(path).read()
newtext = sed_replace(search, replace, text)
file(path, 'w').write(newtext)

...
infile_sed_replace('RunMin=[0-9][0-9]*', 'RunMin=%s' % k, runFile)
...

> 
> You'll notice, the last subprocess call is commented out. Right now I'm
> just getting to that point to make sure everything is working. So, it
> seems to work, but I'm not sure how to get it to work if I change the
> command to nohup. I still want python to wait for it to return, in fact,
> I would like to set the python job running in the background as well...
> so what I'm looking at doing is:
>  
> % nohup myControl.py
>---> which will make several subprocess.call(s) including some that
> should be 'nohupped' as well...

This strikes me as nohup abuse, though I'm not entirely certain what you
are trying to accomplish. Since you plan to nohup *and* background your
script anyway, you might be better served by preemptively detaching from
the controlling terminal (or daemonizing) instead. If so, you may find
one or both of the following recipes useful:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278731
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012

Of course, you may need to handle logging a little differently.

HTH,
Marty
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess help, nohup

[John]

Hello,

I've written a script which conducts several subprocess calls and then
ultimately calls a shell script which runs even more programs... my script
is using subprocess to execute a few sed calls, and then execute the script.
I'm getting strange behavior:

Here's a snippet of my script (any general comments are also welcome):

   if RUNS[k][3]==1: #model run complete, run is a DICT
compWebPages=self.getCompletedWeb() #returns a DICT, which
RUNS[k][0] COULD be a key
if RUNS[k][0] not in compWebPages.keys():
 runFile='make_'+RUNS[k][0]
 cmd = """sed -e 's/RUNmin=[0-9][0-9]*/RUNmin=%s/g' %s  > jnk""" %
(k,'make_wwwpages');
 subprocess.call(cmd,shell=True)
 cmd = """cat jnk | sed -e 's/RUNmax=[0-9][0-9]*/RUNmax=%s/g' > %s""" %
(k,runFile);
 subprocess.call(cmd,shell=True); subprocess.call('rm jnk',shell=True);
 os.chmod(runFile,0744)
 cmd="""./%s""" % (runFile)
 print "Starting %s" % (runFile)
 #subprocess.call(cmd,shell=True)
 q=raw_input('continue?');
 print "Done with: %s" % (RUNS[k][0])
 cmd="""rm %s""" % (runFile); subprocess.call(cmd,shell=True)


You'll notice, the last subprocess call is commented out. Right now I'm just
getting to that point to make sure everything is working. So, it seems to
work, but I'm not sure how to get it to work if I change the command to
nohup. I still want python to wait for it to return, in fact, I would like
to set the python job running in the background as well... so what I'm
looking at doing is:

% nohup myControl.py
   ---> which will make several subprocess.call(s) including some that
should be 'nohupped' as well...

Suggestions on the syntax of how to do this?

Thanks!






-- 
Configuration
``
Plone 2.5.3-final,
CMF-1.6.4,
Zope (Zope 2.9.7-final, python 2.4.4, linux2),
Five 1.4.1,
Python 2.4.4 (#1, Jul 3 2007, 22:58:17) [GCC 4.1.1 20070105 (Red Hat
4.1.1-51)],
PIL 1.1.6
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess stdout nohup question

[John]

When I run a python script with nohup, my print statements are not being
written to nohup.out, why is that? Should nohup.out capture all stdout
statements?

-- 
Configuration
``
Plone 2.5.3-final,
CMF-1.6.4,
Zope (Zope 2.9.7-final, python 2.4.4, linux2),
Five 1.4.1,
Python 2.4.4 (#1, Jul 3 2007, 22:58:17) [GCC 4.1.1 20070105 (Red Hat
4.1.1-51)],
PIL 1.1.6
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and su

[Michael Meier]

> how I thought that it was supposed to work was it would allow me to use
> sbp.communicate() to
> send stuff to the stdin, and get information out. What do get is a prompt
> ask for my password. 

I believe that su does not read its input from stdin but from its
controlling tty. So you'd have to open a pseudo terminal: have a look at
the pty module.

cheers,
Michael

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and su

[Alan Gauld]

"Alan Gauld" <[EMAIL PROTECTED]> wrote

> Oddly I just tried some experiments and I can't get subprocess
> to work with the stdout=PIPE option, it used to...
>
> Anyone have any idea what my obvious mistake is?
>
> Traceback (most recent call last):
>  File "", line 1, in ?
>  File "C:\Python24\lib\subprocess.py", line 533, in __init__
>(p2cread, p2cwrite,
>  File "C:\Python24\lib\subprocess.py", line 593, in _get_handles
>p2cread = self._make_inheritable(p2cread)
>  File "C:\Python24\lib\subprocess.py", line 634, in 
> _make_inheritable
>DUPLICATE_SAME_ACCESS)
> TypeError: an integer is required

Its a PyCrust issue, the code works fine in any of my other shells.
Not sure why PyCrust barfs but it does. I might try posting on
the wxPython list to see if I get an answer...

Alan G 


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and su

[Alan Gauld]

"Tino Dai" <[EMAIL PROTECTED]> wrote

> PIPE=subprocess.pipe

Not sure why you need this?

> sbp=subprocess.Popen
> (["su","-",stdin=PIPE,stdout=PIPE,close_fds=True,shell=True)

and I'm not sure what that [ is doing in there? I assume you meant
to have a closing ] after the "-"?

Have you looked at the examples in the module documents?
They cover most common usages.

> sbp.communicate() to send stuff to the stdin, and get information
> out. What do get is a prompt ask for my password.

su may be spitting the prompt out to stderr. This is quite common
for Unix programs and you might have to assign Popen.stderr to PIPE
and read that too.

Otherwise you seem to be on the right track.

Oddly I just tried some experiments and I can't get subprocess
to work with the stdout=PIPE option, it used to...

Anyone have any idea what my obvious mistake is?


import subprocess as sub
po = sub.Popen('ls -l', shell=True, stdout=sub.PIPE)

gives:

Traceback (most recent call last):
  File "", line 1, in ?
  File "C:\Python24\lib\subprocess.py", line 533, in __init__
(p2cread, p2cwrite,
  File "C:\Python24\lib\subprocess.py", line 593, in _get_handles
p2cread = self._make_inheritable(p2cread)
  File "C:\Python24\lib\subprocess.py", line 634, in _make_inheritable
DUPLICATE_SAME_ACCESS)
TypeError: an integer is required
--

It works without the stdout argument.

Puzzled...

Alan G 


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and su

[Eric Brunson]

Tino Dai wrote:
> Hi there,
>
>  I'm have a tough time figuring out how to get su and subprocess 
> working. I have
>
> PIPE=subprocess.pipe
>
> sbp=subprocess.Popen(["su","-",stdin=PIPE,stdout=PIPE,close_fds=True,shell=True)
>  
>
>
> how I thought that it was supposed to work was it would allow me to 
> use sbp.communicate() to
> send stuff to the stdin, and get information out. What do get is a 
> prompt ask for my password. 

I think you have the right understanding of Popen, you seem to be 
missing out on "su".  What happens when you type "su -" on the command line?

> Does
> anybody have an example that I can see (I have already check most of 
> the entries in google( or should
> I just use pexpect instead?
>
> -Thanks,
> Tino
>
> 
>
> ___
> Tutor maillist  -  Tutor@python.org
> http://mail.python.org/mailman/listinfo/tutor
>   

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess and su

[Tino Dai]

Hi there,

 I'm have a tough time figuring out how to get su and subprocess
working. I have

PIPE=subprocess.pipe

sbp=subprocess.Popen
(["su","-",stdin=PIPE,stdout=PIPE,close_fds=True,shell=True)

how I thought that it was supposed to work was it would allow me to use
sbp.communicate() to
send stuff to the stdin, and get information out. What do get is a prompt
ask for my password. Does
anybody have an example that I can see (I have already check most of the
entries in google( or should
I just use pexpect instead?

-Thanks,
Tino
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and signals

[Noufal Ibrahim]

Eric Brunson wrote:
> Noufal Ibrahim wrote:
[..]
>> def createSignalDelegator(p):
>>  def sighandler(signal,frame,pmake = p):
>>  os.kill(pmake.pid,signal)
>>  return sighandler
>>
>> pmake = subprocess.Popen(pmake_cmd, bufsize = 1, stdout = 
>> subprocess.PIPE, stderr = subprocess.STDOUT)
>>
>> signal.signal(signal.SIGINT,createSignalDelegator(pmake))
>> signal.signal(signal.SIGQUIT,createSignalDelegator(pmake))
>> for op in pmake.stdout:
>> print "-- %s"%str(op).strip()
>> --
>>
[..]
> I'm not an expert on the subprocess module, but I've used it a bit and 
> I'm referring to the docs as I write this.
> 
>> 1. I don't see any output (from my 'build') on screen when I run my 
>> wrapper.
>>   
> 
> Why do you think you should?  You've asked subprocess to pipe its output 
> you your parent process, I believe you would need to use the 
> communicate() method to get the output generated by the subprocess.

If I read from the stdout attribute of the Popen object, I should get 
the output of the pipe. Shouldn't I? That's what I'm trying to do. I 
think the code above is broken though so I've changed the last two lines 
of the snippet to look like this

line = pmake.stdout.readline()
while line:
 print "-- %s"%str(line).strip()
 line = pmake.stdout.readline()

That sounds more sensible. The docs for communicate warn against using 
it when the data size is large and that's the case here. I did however 
try it like so
(stdout,dummy) = pmake.communicate()
line = stdout.readline()
while line:
print "-- %s"%str(line).strip()
line = stdout.readline()

and I get similar behaviour.

>> 3. I sometimes get a pipe error and the whole wrapper dies leaving the 
>> other program running.
>>   
> 
> Exact error messages help a lot.  :-)

Yeah I know. I didn't have access to the machine where I got the 
original error so I thought I'd slur around there a bit. :)

But I got it now (this is with the original snippet).

Traceback (most recent call last):
   File "./build_wrapper.py", line 246, in 
 main()
   File "./build_wrapper.py", line 231, in main
 run_pmake(target,options,extra_flags,pmake_options)
   File "./build_wrapper.py", line 209, in run_pmake
 line = pmake.stdout.readline()
IOError: [Errno 4] Interrupted system call


This is with python2.4 (if that's useful). Also, the output of the 
actual program will be a bit slow (since it's doing compiles and stuff). 
Will some intermediate buffering create any trouble?

Thanks.


-- 
~noufal
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and signals

[Eric Brunson]

Noufal Ibrahim wrote:
> Hello everyone,
>I've come across a situation which is somewhat confusing to me.
>I googled for some details and came across another email thread on 
> this very list but couldn't really glean a solution out of it.
>
>I have a program (a compiled binary) for which I need to write a 
> wrapper (in python). The wrapper will construct some sane command line 
> defaults for this binary and then execute it while storing some 
> statistics like who launched it, where and when.
>
>Now this program will continuously print output (it's a parallel 
> version of make). It it receives a SIGINT (via a Ctrl-C), it will print 
> some statistics and go on with the build. If it receives a Ctrl-\ 
> (SIGQUIT I think), it will terminate. I want my wrapper to be able to 
> read the output of this program and print it while allowing these two 
> signals to be passed to it.
>
>My wrapper (let's call it wrapper.py) has something like this
>
> --
> def createSignalDelegator(p):
>  def sighandler(signal,frame,pmake = p):
>  os.kill(pmake.pid,signal)
>  return sighandler
>
> pmake = subprocess.Popen(pmake_cmd, bufsize = 1, stdout = 
> subprocess.PIPE, stderr = subprocess.STDOUT)
>
> signal.signal(signal.SIGINT,createSignalDelegator(pmake))
> signal.signal(signal.SIGQUIT,createSignalDelegator(pmake))
> for op in pmake.stdout:
> print "-- %s"%str(op).strip()
> --
>
> I've substituted my actual binary with a simple python script that 
> continuously prints a string and which traps sigint and sigquit to 
> appear like the binary. It seems to be receiving the signals fine (since 
> I log it's activity into a separate file) but the problems I get are like so
>   

I'm not an expert on the subprocess module, but I've used it a bit and 
I'm referring to the docs as I write this.

> 1. I don't see any output (from my 'build') on screen when I run my 
> wrapper.
>   

Why do you think you should?  You've asked subprocess to pipe its output 
you your parent process, I believe you would need to use the 
communicate() method to get the output generated by the subprocess.


> 2. When I send a ^C to the wrapper, I don't see the output from the 
> build script.
>   

Fix #1 first and then this should fall into place.

> 3. I sometimes get a pipe error and the whole wrapper dies leaving the 
> other program running.
>   

Exact error messages help a lot.  :-)

> I'd appreciate any insights into the problem. I'm not sure where to 
> start looking.
>
> Thanks.
>
>
>   

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess and signals

[Noufal Ibrahim]

Hello everyone,
   I've come across a situation which is somewhat confusing to me.
   I googled for some details and came across another email thread on 
this very list but couldn't really glean a solution out of it.

   I have a program (a compiled binary) for which I need to write a 
wrapper (in python). The wrapper will construct some sane command line 
defaults for this binary and then execute it while storing some 
statistics like who launched it, where and when.

   Now this program will continuously print output (it's a parallel 
version of make). It it receives a SIGINT (via a Ctrl-C), it will print 
some statistics and go on with the build. If it receives a Ctrl-\ 
(SIGQUIT I think), it will terminate. I want my wrapper to be able to 
read the output of this program and print it while allowing these two 
signals to be passed to it.

   My wrapper (let's call it wrapper.py) has something like this

--
def createSignalDelegator(p):
 def sighandler(signal,frame,pmake = p):
 os.kill(pmake.pid,signal)
 return sighandler

pmake = subprocess.Popen(pmake_cmd, bufsize = 1, stdout = 
subprocess.PIPE, stderr = subprocess.STDOUT)

signal.signal(signal.SIGINT,createSignalDelegator(pmake))
signal.signal(signal.SIGQUIT,createSignalDelegator(pmake))
for op in pmake.stdout:
print "-- %s"%str(op).strip()
--

I've substituted my actual binary with a simple python script that 
continuously prints a string and which traps sigint and sigquit to 
appear like the binary. It seems to be receiving the signals fine (since 
I log it's activity into a separate file) but the problems I get are like so

1. I don't see any output (from my 'build') on screen when I run my 
wrapper.
2. When I send a ^C to the wrapper, I don't see the output from the 
build script.
3. I sometimes get a pipe error and the whole wrapper dies leaving the 
other program running.

I'd appreciate any insights into the problem. I'm not sure where to 
start looking.

Thanks.


-- 
~noufal
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and launching an editor

[Alan Gauld]

"Jason Massey" <[EMAIL PROTECTED]> wrote

> gedit launches with a blank file called window1.  The python shell 
> is now
> waiting for gedit to exit before it does anything else.
>
> 2nd console
> ---
import subprocess
subprocess.call('gedit --new-window window2',shell=True)
>
> Another instance of gedit launches with a blank file called window2. 
> The
> difference here is that the 2nd python shell instantly returns an 
> exit code
> of 0.

Interesting. I suspect this sets a second thread of the same process
off. Can you do a ps to see how many gedit processes are running?

The Gedit Python link might be worth investigatying if you insist
on using gedit. But then your solution might not work for anyone
else... I do with apps wouldn't try to be too clever.

Alan G.



___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and launching an editor

[Alan Gauld]

"Brian van den Broek" <[EMAIL PROTECTED]> wrote

> gedit is the default text editor on my ubuntu feisty system, so in 
> the
> first instance, I've tried to do this with gedit. The 
> subprocess.call:
>
> >>> subprocess.call("gedit somefilename", shell=True)
>

You should probably check the VISUAL and EDITOR environment
settings to find out the users preferred editor. Traditionally VISUAL
outguns EDITOR...

> works just fine *provided* that no instance of gedit is running when 
> I
> invoke .call.

That must be a feature of gedit that it only allows one instance
of gedit to run at a time.

What happens if you try to invoke two instances from an OS prompt?

> Interestingly, it works just fine if I use emacs in place of gedit,

Yep, emacs is happy with multiple instances (or can use emacs
server to ensure multiple calls go to the same client to save 
resources)

> Is there any way to get it to work with gedit as it
> is with emacs?

Change gedit I suspect. At least its open source!

> (In particular, I've no clue why gedit and emacs behave differently 
> in

Almost certainly this is behaviour built into the application.
subprocess.call is pretty much a straight equivalent to os.system
and just returns whatever the app returns...

-- 
Alan Gauld
Author of the Learn to Program web site
http://www.freenetpages.co.uk/hp/alan.gauld 


___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess and launching an editor

[Jason Massey]

I'm running Feisty as well.

I launched python in two separate consoles and tried this:

1st console


import subprocess
subprocess.call('gedit --new-window window1',shell=True)


gedit launches with a blank file called window1.  The python shell is now
waiting for gedit to exit before it does anything else.

2nd console
---

import subprocess
subprocess.call('gedit --new-window window2',shell=True)


Another instance of gedit launches with a blank file called window2.  The
difference here is that the 2nd python shell instantly returns an exit code
of 0.

So you can definately launch multiple instances, but I'm not sure how you
would determine when the user was done editing the file after the first
instance of gedit has been launched.

Perhaps you could use this:

http://live.gnome.org/Gedit/PythonPluginHowTo

If you're going to stick with gedit.

On 6/22/07, Brian van den Broek <[EMAIL PROTECTED]> wrote:


Hi all,

I want to have a script launch an editor open to a particular file and
wait until that editor has closed before continuing. The aim is to
allow the user to make edits to the file, have to script know that the
edits are completed, and then make use of the newly saved file contents.

gedit is the default text editor on my ubuntu feisty system, so in the
first instance, I've tried to do this with gedit. The subprocess.call:

>>> subprocess.call("gedit somefilename", shell=True)

works just fine *provided* that no instance of gedit is running when I
invoke .call. However, if there is a gedit window up and running
(there usually is on my system ;-), the .call immediately returns exit
status 0:

>>> subprocess.Popen("ps -e|grep gedit", shell=True)

>>> 26066 pts/200:00:01 gedit

>>> subprocess.call("gedit somefilename", shell=True)
0
>>> # The exit code is produced instantaneously

Interestingly, it works just fine if I use emacs in place of gedit,
irrespective of whether emacs was running before the subprocess.call
invocation or not. Is there any way to get it to work with gedit as it
is with emacs?

I am largely ignorant of the family of modules which subprocess was
designed to replace, and also of the details of processes on linux.
(In particular, I've no clue why gedit and emacs behave differently in
this respect.)

Thanks and best,

Brian vdB
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess and launching an editor

[Brian van den Broek]

Hi all,

I want to have a script launch an editor open to a particular file and 
wait until that editor has closed before continuing. The aim is to 
allow the user to make edits to the file, have to script know that the 
edits are completed, and then make use of the newly saved file contents.

gedit is the default text editor on my ubuntu feisty system, so in the 
first instance, I've tried to do this with gedit. The subprocess.call:

 >>> subprocess.call("gedit somefilename", shell=True)

works just fine *provided* that no instance of gedit is running when I 
invoke .call. However, if there is a gedit window up and running 
(there usually is on my system ;-), the .call immediately returns exit 
status 0:

 >>> subprocess.Popen("ps -e|grep gedit", shell=True)

 >>> 26066 pts/200:00:01 gedit

 >>> subprocess.call("gedit somefilename", shell=True)
0
 >>> # The exit code is produced instantaneously

Interestingly, it works just fine if I use emacs in place of gedit, 
irrespective of whether emacs was running before the subprocess.call 
invocation or not. Is there any way to get it to work with gedit as it 
is with emacs?

I am largely ignorant of the family of modules which subprocess was 
designed to replace, and also of the details of processes on linux. 
(In particular, I've no clue why gedit and emacs behave differently in 
this respect.)

Thanks and best,

Brian vdB
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Subprocess popen problem depending on how I start program

[pytutor . 20 . 247ob]

Hi,
I'm writing a program that downloads audio streams from the net, to do
this I use mplayer and tcpdump, which I kick off as detached processes
(if thats the correct terminology), I do it this way because i'm using
pygtk and I dont want the program to wait for the processes to finish
otherwise the app would appear to freeze.
I have written 2 modules, one contains a class and methods that do all
the work, the other is the gui module, I've got it working fairly well
at least when I start the program from within a bash shell (Im using
Ubuntu Linux BTW) The modules both have a line '#!/usr/bin/env python'
at the top and are set as executable too so I can run them from a bash
shell.
Now I have one eye on distributing the modules in the future I'm
thinking how people will start the app, and I discover that if I use
the 'run application' program (alt F2 in most linux's I think) to call
the module it starts ok and works normally until I attempt to use the
part that starts the subprocess in the background, it just does not
work and the gui continues to respond as if nothing ever happened. When
I check the processes running I can see that it has not started a sub
shell to run the command. also tried other things like creating an
application launcher, this will work if I tick 'run in terminal' but
not otherwise.
I've also started learning about distutils and installed the modules on
my system using it, then created a python script (made it executable)
and tried using this to start the program, same problem.

I'm scratching around trying to find an answer but I'm at a loss now so
would sure appreciate some help, thanks.

Wayne..
ps. sorry if it's wordy, I want to give enough info. if you need
anymore info just say.





___ 
New Yahoo! Mail is the ultimate force in competitive emailing. Find out more at 
the Yahoo! Mail Championships. Plus: play games and win prizes. 
http://uk.rd.yahoo.com/evt=44106/*http://mail.yahoo.net/uk 
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess & pyw conflict ?

[Dave S]

On Thursday 07 December 2006 10:25, Dave S wrote:
> On Thursday 07 December 2006 00:31, Luke Paireepinart wrote:
> > Dave S wrote:
> > > Hi all,
> > >
> > > I thought I had my solution with subprocess ... my test code ...
> > >
> > >
> > >
> > > #!/usr/bin/env python
> > > # -*- coding: iso8859_1 -*-
> > >
> > > import subprocess
> > >
> > > a = subprocess.Popen('tasklist.exe', bufsize=0, stdout=subprocess.PIPE,
> > > universal_newlines=True)
> > > op = a.stdout.readlines()
> > >
> > > for i in op:
> > > if i.split(' ')[0] == 'gmanager.exe':
> > > f = open('E:\Documents and Settings\All
> > > Users\Desktop\gsr_running', 'w')
> > > f.close()
> > >
> > >
> > >
> > > works a treat when I run it as proc.py detects the process I am looking
> > > for & writes a dummy file to the desktop. :) but I get a black windows
> > > terminal flash up.
> > >
> > > The code will eventually run in an app.pyw so to check it would be OK I
> > > renamed my working proc.py to proc.pyw - it fails :(, No window (as
> > > expected), no dummy file (not expected) - the process gmanager.exe is
> > > running.
> > >
> > > So there seems to be a problem with subprocess & pyw
> > >
> > > Googling I found ...
> > > https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1358527&gr
> > >ou p_id=5470 So I tried the suggested workaround with proc.pyw ...
> > >
> > >
> > >
> > > #!/usr/bin/env python
> > > # -*- coding: iso8859_1 -*-
> > >
> > > import subprocess
> > >
> > > a = subprocess.Popen('tasklist.exe', bufsize=0, stdin=subprocess.PIPE,
> > > stdout=subprocess.PIPE, universal_newlines=True)
> > > op = a.stdout.readlines()
> > > a.stdin.close()
> > >
> > > for i in op:
> > > if i.split(' ')[0] == 'gmanager.exe':
> > > f = open('E:\Documents and Settings\All
> > > Users\Desktop\gsr_running', 'w')
> > > f.close()
> > >
> > >
> > > Still zip, and because there is no terminal, I cannot view any errors !
> > >
> > > Any suggestions welcome :)
> >
> > Well, because I'm batting 0 on this thread so far, I think I'll just go
> > ahead and suggest another bad solution!
> > You could try redirecting the output of sys.stderr to a file, and you
> > might be able to see the error message! :D
> >
> > If you hadn't said 'any suggestions welcome' I might've kept this to
> > myself :P
>
> Ahhh .. 'any suggestions' is an SOS call - all any any ideas are warmly
> greeted :)
>
> >  >>> import sys
> >  >>> class TestClass(object):
> >
> > def __init__(self):
> > self.data = []
> > def write(self,item):
> > self.data.append(item)
> >
> >  >>> a = TestClass()
> >  >>> sys.stderr = a
> >  >>> salfjdsljfka321423
> >  >>> print a.data
> >
> > ['\nTraceback (most recent call last):', '\n', '  File "",
> > line 1, in -toplevel-\n', 'salfjdsljfka321423\n', "NameError: name
> > 'salfjdsljfka321423' is not defined\n"]
> >
> > Except instead of a file-like class, you could just use a real file.
> > But then it would only leave the last line intact.
> > So you'd probably want to make a class that wraps a file object, where
> > the write method just appends to an internal list,
> > and it writes it all out to the file when you call the Class.close() or
> > whatever.
> > Actually, I guess the program stops executing on an exception...
> > Hmm, not really sure what you'd do exactly.
> >
> >
> > Sure, there are better solutions, and this doesn't really help you with
> > your original problem, but it at least lets you see your error message!
> > HTH,
> > -Luke
>
> Thanks for that - I will give it a go & post back :)
>


Oh my head  OK after much tinkering I got the following to work with .pyw


# scan windows task list to see if GSR is running
f = os.popen('tasklist.exe', 'r')
plist = f.readlines()
f.close

gsr_running = False  # scan for GSR program
for line in plist:
if line.split(' ')[0] == 'gmanager.exe': gsr_running = True

A 

Dave

can relax, chill, project finished ... all works ... god .. (homer 
simpson moment !)






> > > ___
> > > Tutor maillist  -  Tutor@python.org
> > > http://mail.python.org/mailman/listinfo/tutor
>
> ___
> Tutor maillist  -  Tutor@python.org
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess & pyw conflict ?

[Dave S]

On Thursday 07 December 2006 00:31, Luke Paireepinart wrote:
> Dave S wrote:
> > Hi all,
> >
> > I thought I had my solution with subprocess ... my test code ...
> >
> >
> >
> > #!/usr/bin/env python
> > # -*- coding: iso8859_1 -*-
> >
> > import subprocess
> >
> > a = subprocess.Popen('tasklist.exe', bufsize=0, stdout=subprocess.PIPE,
> > universal_newlines=True)
> > op = a.stdout.readlines()
> >
> > for i in op:
> > if i.split(' ')[0] == 'gmanager.exe':
> > f = open('E:\Documents and Settings\All
> > Users\Desktop\gsr_running', 'w')
> > f.close()
> >
> >
> >
> > works a treat when I run it as proc.py detects the process I am looking
> > for & writes a dummy file to the desktop. :) but I get a black windows
> > terminal flash up.
> >
> > The code will eventually run in an app.pyw so to check it would be OK I
> > renamed my working proc.py to proc.pyw - it fails :(, No window (as
> > expected), no dummy file (not expected) - the process gmanager.exe is
> > running.
> >
> > So there seems to be a problem with subprocess & pyw
> >
> > Googling I found ...
> > https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1358527&grou
> >p_id=5470 So I tried the suggested workaround with proc.pyw ...
> >
> >
> >
> > #!/usr/bin/env python
> > # -*- coding: iso8859_1 -*-
> >
> > import subprocess
> >
> > a = subprocess.Popen('tasklist.exe', bufsize=0, stdin=subprocess.PIPE,
> > stdout=subprocess.PIPE, universal_newlines=True)
> > op = a.stdout.readlines()
> > a.stdin.close()
> >
> > for i in op:
> > if i.split(' ')[0] == 'gmanager.exe':
> > f = open('E:\Documents and Settings\All
> > Users\Desktop\gsr_running', 'w')
> > f.close()
> >
> >
> > Still zip, and because there is no terminal, I cannot view any errors !
> >
> > Any suggestions welcome :)
>
> Well, because I'm batting 0 on this thread so far, I think I'll just go
> ahead and suggest another bad solution!
> You could try redirecting the output of sys.stderr to a file, and you
> might be able to see the error message! :D
>
> If you hadn't said 'any suggestions welcome' I might've kept this to
> myself :P

Ahhh .. 'any suggestions' is an SOS call - all any any ideas are warmly 
greeted :)


>
>  >>> import sys
>  >>> class TestClass(object):
>
> def __init__(self):
> self.data = []
> def write(self,item):
> self.data.append(item)
>
>  >>> a = TestClass()
>  >>> sys.stderr = a
>  >>> salfjdsljfka321423
>  >>> print a.data
>
> ['\nTraceback (most recent call last):', '\n', '  File "",
> line 1, in -toplevel-\n', 'salfjdsljfka321423\n', "NameError: name
> 'salfjdsljfka321423' is not defined\n"]
>
> Except instead of a file-like class, you could just use a real file.
> But then it would only leave the last line intact.
> So you'd probably want to make a class that wraps a file object, where
> the write method just appends to an internal list,
> and it writes it all out to the file when you call the Class.close() or
> whatever.
> Actually, I guess the program stops executing on an exception...
> Hmm, not really sure what you'd do exactly.
>
>
> Sure, there are better solutions, and this doesn't really help you with
> your original problem, but it at least lets you see your error message!
> HTH,
> -Luke

Thanks for that - I will give it a go & post back :)

>
> > ___
> > Tutor maillist  -  Tutor@python.org
> > http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] subprocess & pyw conflict ?

[Luke Paireepinart]

Dave S wrote:
> Hi all,
>
> I thought I had my solution with subprocess ... my test code ...
>
>
>
> #!/usr/bin/env python
> # -*- coding: iso8859_1 -*- 
>
> import subprocess
> 
> a = subprocess.Popen('tasklist.exe', bufsize=0, stdout=subprocess.PIPE, 
> universal_newlines=True)
> op = a.stdout.readlines()
>
> for i in op:
> if i.split(' ')[0] == 'gmanager.exe':
> f = open('E:\Documents and Settings\All 
> Users\Desktop\gsr_running', 'w')
> f.close()
> 
>
>
> works a treat when I run it as proc.py detects the process I am looking for & 
> writes a dummy file to the desktop. :) but I get a black windows terminal 
> flash up.
>
> The code will eventually run in an app.pyw so to check it would be OK I 
> renamed my working proc.py to proc.pyw - it fails :(, No window (as 
> expected), no dummy file (not expected) - the process gmanager.exe is 
> running.
>
> So there seems to be a problem with subprocess & pyw
>
> Googling I found ...
> https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1358527&group_id=5470
> So I tried the suggested workaround with proc.pyw ...
>
>
>
> #!/usr/bin/env python
> # -*- coding: iso8859_1 -*- 
>
> import subprocess
> 
> a = subprocess.Popen('tasklist.exe', bufsize=0, stdin=subprocess.PIPE, 
> stdout=subprocess.PIPE, universal_newlines=True)
> op = a.stdout.readlines()
> a.stdin.close()
>
> for i in op:
> if i.split(' ')[0] == 'gmanager.exe':
> f = open('E:\Documents and Settings\All 
> Users\Desktop\gsr_running', 'w')
> f.close()
> 
>
> Still zip, and because there is no terminal, I cannot view any errors !
>
> Any suggestions welcome :)
>   
Well, because I'm batting 0 on this thread so far, I think I'll just go 
ahead and suggest another bad solution!
You could try redirecting the output of sys.stderr to a file, and you 
might be able to see the error message! :D

If you hadn't said 'any suggestions welcome' I might've kept this to 
myself :P

 >>> import sys
 >>> class TestClass(object):
def __init__(self):
self.data = []
def write(self,item):
self.data.append(item)
 >>> a = TestClass()
 >>> sys.stderr = a
 >>> salfjdsljfka321423
 >>> print a.data
['\nTraceback (most recent call last):', '\n', '  File "", 
line 1, in -toplevel-\n', 'salfjdsljfka321423\n', "NameError: name 
'salfjdsljfka321423' is not defined\n"]

Except instead of a file-like class, you could just use a real file.  
But then it would only leave the last line intact.
So you'd probably want to make a class that wraps a file object, where 
the write method just appends to an internal list,
and it writes it all out to the file when you call the Class.close() or 
whatever.
Actually, I guess the program stops executing on an exception...
Hmm, not really sure what you'd do exactly.


Sure, there are better solutions, and this doesn't really help you with 
your original problem, but it at least lets you see your error message!
HTH,
-Luke
>
> ___
> Tutor maillist  -  Tutor@python.org
> http://mail.python.org/mailman/listinfo/tutor
>
>   

___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor

[Tutor] subprocess & pyw conflict ?

[Dave S]

Hi all,

I thought I had my solution with subprocess ... my test code ...



#!/usr/bin/env python
# -*- coding: iso8859_1 -*- 

import subprocess

a = subprocess.Popen('tasklist.exe', bufsize=0, stdout=subprocess.PIPE, 
universal_newlines=True)
op = a.stdout.readlines()

for i in op:
if i.split(' ')[0] == 'gmanager.exe':
f = open('E:\Documents and Settings\All 
Users\Desktop\gsr_running', 'w')
f.close()



works a treat when I run it as proc.py detects the process I am looking for & 
writes a dummy file to the desktop. :) but I get a black windows terminal 
flash up.

The code will eventually run in an app.pyw so to check it would be OK I 
renamed my working proc.py to proc.pyw - it fails :(, No window (as 
expected), no dummy file (not expected) - the process gmanager.exe is 
running.

So there seems to be a problem with subprocess & pyw

Googling I found ...
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1358527&group_id=5470
So I tried the suggested workaround with proc.pyw ...



#!/usr/bin/env python
# -*- coding: iso8859_1 -*- 

import subprocess

a = subprocess.Popen('tasklist.exe', bufsize=0, stdin=subprocess.PIPE, 
stdout=subprocess.PIPE, universal_newlines=True)
op = a.stdout.readlines()
a.stdin.close()

for i in op:
if i.split(' ')[0] == 'gmanager.exe':
f = open('E:\Documents and Settings\All 
Users\Desktop\gsr_running', 'w')
f.close()


Still zip, and because there is no terminal, I cannot view any errors !

Any suggestions welcome :)

Dave



___
Tutor maillist  -  Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor