Re: subprocess module: execution of standard binaries without shell?
On Thu, Feb 26, 2009 at 2:41 AM, Visco Shaun wrote: > hi all > > while getting used to with subprocess module i failed in executuing a) > but succeeded in running b). Can anyone explain me why as i am providing > absolute path? Is this has to do anything with shared library.. which > must be accessed based on system variables? > > > a) pipe = subprocess.Popen("/bin/ls /", stdout=subprocess.PIPE, > close_fds=True) > ==>OSError: [Errno 2] No such file or directory You need to use a list of arguments, not just a string. You're currently telling Python to try and run a nonexistent directory (specifically, the "ls " subdirectory of /bin), since the string way of calling Popen assumes that the *entire* string is the path to the executable when shell=False. The correct way is to provide the path to the binary and then each of its arguments, in a list: pipe = subprocess.Popen(["/bin/ls", "/"], stdout=subprocess.PIPE, close_fds=True) > b) pipe = subprocess.Popen("/bin/ls /", stdout=subprocess.PIPE, > close_fds=True, shell=True) This works because shell=True sends the string through the shell, which tokenizes it and runs it, effectively splitting the string into a list for you. However, shell=True is dangerous as you need to be careful to escape special characters, whereas that's not necessary for the 'shell=False and list' way of calling Popen. Cheers, Chris -- Follow the path of the Iguana... http://rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: subprocess module: execution of standard binaries without shell?
Visco Shaun schrieb: > hi all > > while getting used to with subprocess module i failed in executuing a) > but succeeded in running b). Can anyone explain me why as i am providing > absolute path? Is this has to do anything with shared library.. which > must be accessed based on system variables? > > > a) pipe = subprocess.Popen("/bin/ls /", stdout=subprocess.PIPE, > close_fds=True) > ==>OSError: [Errno 2] No such file or directory You have to use a list instead of a string here. pipe = subprocess.Popen(["/bin/ls", "/"], stdout=subprocess.PIPE) -- http://mail.python.org/mailman/listinfo/python-list
subprocess module: execution of standard binaries without shell?
hi all while getting used to with subprocess module i failed in executuing a) but succeeded in running b). Can anyone explain me why as i am providing absolute path? Is this has to do anything with shared library.. which must be accessed based on system variables? a) pipe = subprocess.Popen("/bin/ls /", stdout=subprocess.PIPE, close_fds=True) ==>OSError: [Errno 2] No such file or directory b) pipe = subprocess.Popen("/bin/ls /", stdout=subprocess.PIPE, close_fds=True, shell=True) -- Thanks & Regards visco -- http://mail.python.org/mailman/listinfo/python-list
Re: Subprocess with and without shell
George Sakkis <[EMAIL PROTECTED]> wrote: > On May 15, 5:30 am, Nick Craig-Wood <[EMAIL PROTECTED]> wrote: >> George Sakkis <[EMAIL PROTECTED]> wrote: >>> I'm trying to figure out why Popen captures the stderr of a specific >>> command when it runs through the shell but not without it. IOW: >>> cmd = [my_exe, arg1, arg2, ..., argN] >>> if 1: # this captures both stdout and stderr as expected >>> pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE) >>> else: # this captures only stdout >>> pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE) >>> # this prints the empty string if not run through the shell >>> print "stderr:", pipe.stderr.read() >>> # this prints correctly in both cases >>> print "stdout:", pipe.stdout.read() >>> Any hints ? >> Post an example which replicates the problem! > I would, but the specific executable being spawned is not a python > script, it's a compiled binary (it's not an extension module either; > it's totally unrelated to python). I don't claim there is a bug or > anything suspicious about Popen, but rather I'd like an explanation of > how can a program display different behavior depending on whether it > runs through the shell or not. > George Well, I would try inspecting your environment ... in the shell and from within your Python process. See if there's anything there. If run a command via an interactive shell and it behaves differently when run via Popen then see if perhaps it's doing something like checking to see if it's stdin, or stdout are TTYs (using the C library functions like isatty() for example). You might try running the program under a Pexpect rather than SubProcess (since Pexpect will run the process with it's std* descriptors connected to pty devices). Alternatively try running the program in a shell pipeline to see if it behaves more like you're seeing when you run it under Python. (Since running it in the middle of a pipeline, perhaps with 2>&1 as well, is ensuring that all of the std* descriptors are connected to pipes. (You could also run with 2>/tmp/some.FIFO after doing a mknod p /tmp/some.FIFO (Linux) or mkfifo /tmp/some.FIFO (BSD) to create the named pipe, of course). If none of that worked ... try running the program under stace, truss, ktrace or whatever system call tracing facility your OS provides ... or under gdb. -- Jim Dennis, Starshine: Signed, Sealed, Delivered -- http://mail.python.org/mailman/listinfo/python-list
Re: Subprocess with and without shell
On May 15, 5:30 am, Nick Craig-Wood <[EMAIL PROTECTED]> wrote: > George Sakkis <[EMAIL PROTECTED]> wrote: > > I'm trying to figure out why Popen captures the stderr of a specific > > command when it runs through the shell but not without it. IOW: > > > cmd = [my_exe, arg1, arg2, ..., argN] > > if 1: # this captures both stdout and stderr as expected > > pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE) > > else: # this captures only stdout > > pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE) > > > # this prints the empty string if not run through the shell > > print "stderr:", pipe.stderr.read() > > # this prints correctly in both cases > > print "stdout:", pipe.stdout.read() > > > Any hints ? > > Post an example which replicates the problem! I would, but the specific executable being spawned is not a python script, it's a compiled binary (it's not an extension module either; it's totally unrelated to python). I don't claim there is a bug or anything suspicious about Popen, but rather I'd like an explanation of how can a program display different behavior depending on whether it runs through the shell or not. George -- http://mail.python.org/mailman/listinfo/python-list
Re: Subprocess with and without shell
George Sakkis <[EMAIL PROTECTED]> wrote: > I'm trying to figure out why Popen captures the stderr of a specific > command when it runs through the shell but not without it. IOW: > > cmd = [my_exe, arg1, arg2, ..., argN] > if 1: # this captures both stdout and stderr as expected > pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE) > else: # this captures only stdout > pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE) > > # this prints the empty string if not run through the shell > print "stderr:", pipe.stderr.read() > # this prints correctly in both cases > print "stdout:", pipe.stdout.read() > > Any hints ? Post an example which replicates the problem! My effort works as expected -- z.py #!/usr/bin/python from subprocess import Popen, PIPE cmd = ["./zz.py"] for i in range(2): if i: # this captures both stdout and stderr as expected print "With shell" pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE) else: # this captures only stdout print "Without shell" pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE) # this prints the empty string if not run through the shell print "stderr:", pipe.stderr.read() # this prints correctly in both cases print "stdout:", pipe.stdout.read() ---zz.py #!/usr/bin/python import sys print >>sys.stdout, "Stdout" print >>sys.stderr, "Stderr" Produces $ ./z.py Without shell stderr: Stderr stdout: Stdout With shell stderr: Stderr stdout: Stdout -- Nick Craig-Wood <[EMAIL PROTECTED]> -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Subprocess with and without shell
I'm trying to figure out why Popen captures the stderr of a specific command when it runs through the shell but not without it. IOW: cmd = [my_exe, arg1, arg2, ..., argN] if 1: # this captures both stdout and stderr as expected pipe = Popen(' '.join(cmd), shell=True, stderr=PIPE, stdout=PIPE) else: # this captures only stdout pipe = Popen(cmd, shell=False, stderr=PIPE, stdout=PIPE) # this prints the empty string if not run through the shell print "stderr:", pipe.stderr.read() # this prints correctly in both cases print "stdout:", pipe.stdout.read() Any hints ? George -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
On Friday 10 June 2005 05:30 am, Tomasz Rola wrote: > On Sun, 12 Jun 2005, km wrote: > > > hi all, > > > > can any linux command be invoked/ executed without using shell (bash) ? > > what abt security concerns ? > > Ops, I missed the word "command" when reading your mail for the first > time, and this changes some parts of my previous answer and makes it > shorter: > > There is an execve system call. You don't need neither sh, nor the libc to > run programs. It's described in section 2 of manpages. The rest of the > answer you can get from my previous post. I haven't used it, but according to the Python 2.4 documentation, the subprocess module does not use any shell. Or the shell is python, as it were. Cheers, Terry -- Terry Hancock ( hancock at anansispaceworks.com ) Anansi Spaceworks http://www.anansispaceworks.com -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
Donn Cave wrote: >> Not according the the docs: >> >> Also, for each of these variants, on Unix, cmd may be a >> sequence, in which case arguments will be passed directly to >> the program without shell intervention (as with os.spawnv()). >> If cmd is a string it will be passed to the shell (as with >> os.system()). >> >> It's not exactly clear what "these variants" refer to, but I >> read it as referring to all of the the os.popen functions. >> >> Perhaps it only refers to os.popen[234]? > > Right. The paragraphs seem a little scrambled. Note > the use of "cmd" instead of "command" as the parameter > is named for popen(). Also note "These methods do not > make it possible to retrieve the return code from the > child processes", after the popen() paragraph above tells > you how to do it (using the better term "exit status".) > > Or one may look at the source. FYI, I checked in a little fix to the docs which makes clear what functions the paragraphs pertain to. Also, I changed "return code" to "exit status". Reinhold -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
On 2005-06-10, Donn Cave <[EMAIL PROTECTED]> wrote: >> Also, for each of these variants, on Unix, cmd may be a >> sequence, in which case arguments will be passed directly to >> the program without shell intervention (as with os.spawnv()). >> If cmd is a string it will be passed to the shell (as with >> os.system()). >> >> It's not exactly clear what "these variants" refer to, but I >> read it as referring to all of the the os.popen functions. >> >> Perhaps it only refers to os.popen[234]? > > Right. The paragraphs seem a little scrambled. Note > the use of "cmd" instead of "command" as the parameter > is named for popen(). Also note "These methods do not > make it possible to retrieve the return code from the > child processes", after the popen() paragraph above tells > you how to do it (using the better term "exit status".) > > Or one may look at the source. Or write a 3-line test to see how it really does works. :) -- Grant Edwards grante Yow! ... I don't like at FRANK SINATRA or his visi.comCHILDREN. -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
In article <[EMAIL PROTECTED]>, Grant Edwards <[EMAIL PROTECTED]> wrote: ... > According to the current module reference, that's the behavior > of the os.popen*() functions: > > http://docs.python.org/lib/os-newstreams.html#os-newstreams > > > On UNIX, os.popen is posix.popen, is a simple wrapper around > > the C library popen. It always invokes the shell. > > Not according the the docs: > > Also, for each of these variants, on Unix, cmd may be a > sequence, in which case arguments will be passed directly to > the program without shell intervention (as with os.spawnv()). > If cmd is a string it will be passed to the shell (as with > os.system()). > > It's not exactly clear what "these variants" refer to, but I > read it as referring to all of the the os.popen functions. > > Perhaps it only refers to os.popen[234]? Right. The paragraphs seem a little scrambled. Note the use of "cmd" instead of "command" as the parameter is named for popen(). Also note "These methods do not make it possible to retrieve the return code from the child processes", after the popen() paragraph above tells you how to do it (using the better term "exit status".) Or one may look at the source. Donn Cave, [EMAIL PROTECTED] -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
Donn Cave <[EMAIL PROTECTED]> writes: > In article <[EMAIL PROTECTED]>, > Grant Edwards <[EMAIL PROTECTED]> wrote: > >> On 2005-06-10, Mage <[EMAIL PROTECTED]> wrote: >> >> >>py> file_list = os.popen("ls").read() >> >> >> >>Stores the output of ls into file_list. >> >> >> > These commands invoke shell indeed. >> >> Under Unix, popen will not invoke a shell if it's passed a >> sequence rather than a single string. > > I suspect you're thinking of the popen2 functions. > On UNIX, os.popen is posix.popen, is a simple wrapper > around the C library popen. It always invokes the > shell. > > The no-shell alternatives are spawnv (instead of > system) and the popen2 family (given a sequence > of strings.) Don't forget the one module to rule them all, subprocess: file_list = subprocess.Popen(['ls'], stdout=subprocess.PIPE).communicate()[0] which by default won't use the shell (unless you pass shell=True to it). -- |>|\/|< /--\ |David M. Cooke |cookedm(at)physics(dot)mcmaster(dot)ca -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
On 2005-06-10, Donn Cave <[EMAIL PROTECTED]> wrote: > In article <[EMAIL PROTECTED]>, > Grant Edwards <[EMAIL PROTECTED]> wrote: > >> On 2005-06-10, Mage <[EMAIL PROTECTED]> wrote: >> >> >>py> file_list = os.popen("ls").read() >> >> >> >>Stores the output of ls into file_list. >> >> >> > These commands invoke shell indeed. >> >> Under Unix, popen will not invoke a shell if it's passed a >> sequence rather than a single string. > > I suspect you're thinking of the popen2 functions. According to the current module reference, that's the behavior of the os.popen*() functions: http://docs.python.org/lib/os-newstreams.html#os-newstreams > On UNIX, os.popen is posix.popen, is a simple wrapper around > the C library popen. It always invokes the shell. Not according the the docs: Also, for each of these variants, on Unix, cmd may be a sequence, in which case arguments will be passed directly to the program without shell intervention (as with os.spawnv()). If cmd is a string it will be passed to the shell (as with os.system()). It's not exactly clear what "these variants" refer to, but I read it as referring to all of the the os.popen functions. Perhaps it only refers to os.popen[234]? > The no-shell alternatives are spawnv (instead of system) and > the popen2 family (given a sequence of strings.) > >Donn Cave, [EMAIL PROTECTED] -- Grant Edwards grante Yow! FIRST, I'm covering at you with OLIVE OIL and visi.comPRUNE WHIP!! -- http://mail.python.org/mailman/listinfo/python-list
RE: without shell
KM writes: > can any linux command be invoked/ executed without using shell (bash) ? > what abt security concerns ? Yes. See: http://docs.python.org/lib/module-subprocess.html An exerpt: > On Unix, with shell=False (default): In this case, the Popen class uses > os.execvp() to execute the child program. -- Michael Chermside -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
In article <[EMAIL PROTECTED]>, Grant Edwards <[EMAIL PROTECTED]> wrote: > On 2005-06-10, Mage <[EMAIL PROTECTED]> wrote: > > >>py> file_list = os.popen("ls").read() > >> > >>Stores the output of ls into file_list. > >> > > These commands invoke shell indeed. > > Under Unix, popen will not invoke a shell if it's passed a > sequence rather than a single string. I suspect you're thinking of the popen2 functions. On UNIX, os.popen is posix.popen, is a simple wrapper around the C library popen. It always invokes the shell. The no-shell alternatives are spawnv (instead of system) and the popen2 family (given a sequence of strings.) Donn Cave, [EMAIL PROTECTED] -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
On Fri, 10 Jun 2005 14:13:05 +, Grant Edwards wrote: > On 2005-06-10, Steven D'Aprano <[EMAIL PROTECTED]> wrote: >> On Sun, 12 Jun 2005 23:16:35 +0530, km wrote: >> >>> hi all, >>> >>> can any linux command be invoked/ executed without using shell (bash) ? >> >> py> import os >> py> status = os.system("ls") >> >> Prints the output of ls and stores the exit code into status. > > It's done by invoking the user's SHELL and passing the string > "ls" to it. In the general case, invoking an unknown shell and > passing it a string is fraught with peril. Ah... you learn something new every day. I interpreted the original question as meaning "can Python execute arbitrary Linux commands without exiting the Python interpretor and dropping into a shell prompt?". -- Steven. -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
On 2005-06-10, Mage <[EMAIL PROTECTED]> wrote: >>py> file_list = os.popen("ls").read() >> >>Stores the output of ls into file_list. >> > These commands invoke shell indeed. Under Unix, popen will not invoke a shell if it's passed a sequence rather than a single string. -- Grant Edwards grante Yow! I was in EXCRUCIATING at PAIN until I started visi.comreading JACK AND JILL Magazine!! -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
Steven D'Aprano wrote: >On Sun, 12 Jun 2005 23:16:35 +0530, km wrote: > > > >>hi all, >> >>can any linux command be invoked/ executed without using shell (bash) ? >> >> > >py> import os >py> status = os.system("ls") > >Prints the output of ls and stores the exit code into status. > >py> file_list = os.popen("ls").read() > >Stores the output of ls into file_list. > > > These commands invoke shell indeed. Mage -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
On 2005-06-10, Steven D'Aprano <[EMAIL PROTECTED]> wrote: > On Sun, 12 Jun 2005 23:16:35 +0530, km wrote: > >> hi all, >> >> can any linux command be invoked/ executed without using shell (bash) ? > > py> import os > py> status = os.system("ls") > > Prints the output of ls and stores the exit code into status. It's done by invoking the user's SHELL and passing the string "ls" to it. In the general case, invoking an unknown shell and passing it a string is fraught with peril. > py> file_list = os.popen("ls").read() > > Stores the output of ls into file_list. That also executes a shell (same as os.system()), so it's equally as unreliable and insecure as os.system(). [See the notes at http://docs.python.org/lib/os-newstreams.html#os-newstreams that describe popen.] > Or see the module "commands". > >> what abt security concerns ? > > Yes, there are serious security concerns. You should be *very* careful > about executing strings generated by users. You probably don't want your > program executing something like os.system("rm -rf /"). You've got also got a much better chance of getting what you expect if you don't invoke a shell, but use os.spawn* functions instead. -- Grant Edwards grante Yow! I feel partially at hydrogenated! visi.com -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
On 2005-06-12, km <[EMAIL PROTECTED]> wrote: > can any linux command be invoked/executed without using shell (bash)? Yes -- for some values of "linux command". You can execute anything that's not a bash internal or a bash script without using bash. > what abt security concerns? What about them? -- Grant Edwards grante Yow! I'm young... I'm at HEALTHY... I can HIKE visi.comTHRU CAPT GROGAN'S LUMBAR REGIONS! -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
On Sun, 12 Jun 2005 23:16:35 +0530, km wrote: > hi all, > > can any linux command be invoked/ executed without using shell (bash) ? py> import os py> status = os.system("ls") Prints the output of ls and stores the exit code into status. py> file_list = os.popen("ls").read() Stores the output of ls into file_list. Or see the module "commands". > what abt security concerns ? Yes, there are serious security concerns. You should be *very* careful about executing strings generated by users. You probably don't want your program executing something like os.system("rm -rf /"). -- Steven. -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sun, 12 Jun 2005, km wrote: > hi all, > > can any linux command be invoked/ executed without using shell (bash) ? > what abt security concerns ? Ops, I missed the word "command" when reading your mail for the first time, and this changes some parts of my previous answer and makes it shorter: There is an execve system call. You don't need neither sh, nor the libc to run programs. It's described in section 2 of manpages. The rest of the answer you can get from my previous post. Sorry if I went a bit offtopic in my previous mail. Shouldn't watch tv and write mails at the same time. > regards, > KM Regards, Tomasz Rola - -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:[EMAIL PROTECTED] ** -BEGIN PGP SIGNATURE- Version: PGPfreeware 5.0i for non-commercial use Charset: noconv iQA/AwUBQqlr0RETUsyL9vbiEQIocwCfVh1SsT+RegTaxvNjlsCl8FYupe8AoLH5 qci3LXS1w8bq1ZqH7EKL1HuT =0WoY -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
Re: without shell
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sun, 12 Jun 2005, km wrote: > hi all, > > can any linux command be invoked/ executed without using shell (bash) ? > what abt security concerns ? To answer your question fast, yes it is possible. Just pull every "bad" block from the OS, and put inside some replacement of your own. But it all depends on what exactly you are going to achieve... 1. Disabling rootkits/shellcodes. Without shell (i.e. bash/sh), you loose lots of functionality and you don't get as much in exchange. If what you want really is to disable execution of rootkits, shellcodes etc, then you need to disable almost every interesting program: perl, python, awk, sh, emacs, vi, web browsers with javascript, java, any compiler or interpreter that is installed, and possibly much more but they don't come to my mind right now. After doing so, you get an os that cannot boot past running /sbin/init and is "secure" because it is useless and can be as well turned off. Sure, you can replace/rename all those programs to have functionality and security but this will not protect your computer for too long. It all depends on how much someone wants to get to you. If there is one such person, the above mentioned steps will not help. It also requires much of work and in the result, you will have an incompatible OS i.e., no compatibility beyond some libraries and kernel stuff. I'm not even sure if it is possible to have full KDE/GNOME without shells. The same with X - its startup runs through few shell scripts before the real /usr/bin/X11/X is exec'd. There are better ways of securing Linux with less work and IMHO the resulting OS is much better than anything without shells, etc. at all. Google is your master. www.nsa.gov/selinux/ www.lids.org/ www.openwall.com/ 2. Running some minimal, barebone Linux with carefully carved functionality. You can replace /sbin/init with your own program and make it do whatever you need. Link it statically and you should not even need libraries, just one file and a kernel. Again, sometimes you can get similar or better results without sacrificing the whole OS, and with less work. But this subject is quite broad and so there is not much more to say. > regards, > KM Regards, Tomasz Rola - -- ** A C programmer asked whether computer had Buddha's nature. ** ** As the answer, master did "rm -rif" on the programmer's home** ** directory. And then the C programmer became enlightened... ** ** ** ** Tomasz Rola mailto:[EMAIL PROTECTED] ** -BEGIN PGP SIGNATURE- Version: PGPfreeware 5.0i for non-commercial use Charset: noconv iQA/AwUBQqlqSBETUsyL9vbiEQLVHwCfX3X0IyZLBq3k1uYJElNh1BUOFdIAoKaL ZH5Eqxq2EnN+XpDT9K79FNsK =Jusy -END PGP SIGNATURE- -- http://mail.python.org/mailman/listinfo/python-list
without shell
hi all, can any linux command be invoked/ executed without using shell (bash) ? what abt security concerns ? regards, KM -- http://mail.python.org/mailman/listinfo/python-list