Re: [Tutor] subprocess.call list vs. str argument
> > From: Oscar Benjamin >To: Albert-Jan Roskam >Cc: Dave Angel ; eryksun ; >"Tutor@python.org" >Sent: Wednesday, February 26, 2014 3:38 PM >Subject: Re: [Tutor] subprocess.call list vs. str argument > > >On 26 February 2014 08:50, Albert-Jan Roskam wrote: > >> >> Yesterday evening (it was *late* so forgive me if I wrong) I realized that >> part of my confusion was also caused by the fact that I ran my code in Idle. >> If I called subprocess.call with a list argument, it returned code 0 >> (success) but the generated sphinx code was not the same as when I ran the >> program from the terminal. I concluded that this is some weird interaction >> between Idle (which may also run in a subprocess??) and my own program. I >> also had no problems when I ran (in the terminal): python -c "import >> subprocess; subprocess.call(['sphinx-apidoc'...(etc)])" >> >> I was surprised that the list elements of the argument for subprocess.call >> *have to be* separate tokens, e.g. (earlier in one of Danny's replies): a >> token (list element) '-f -F' won't work, it has to be two separate >> elements/tokens: '-f', '-F'. Apparently, subprocess.call is more than just a >> convenience function that " ".joins the list and escapes the resulting >> string. > >At the operating system level there is no "command line" as such on >posix (Linux, Mac etc.). The command to run a program is a string >giving the name of the program and you can also pass a list of strings >to the program as "arguments". The command line is something that >happens in the shell (e.g. bash). Whereas Python would use quotes and >commas to separate strings in a list of strings the shell just assumes >that whitespace separates strings. > >So in Python you would write > ['qwe', 'asd', 'zxc zxc'] >and it's clear that you have a list of three strings. In the shell >(e.g. bash) you would type > qwe asd "zxc zxc" >and the shell understands that as three separate strings. The quotes >are needed to distinguish it from 4 strings because shell syntax >assumes that spaces separate strings. > >So the idea of the "command line" as a single string that must be >split on whitespace is just something that happens in the shell. It is >supposed to be a convenient way of typing commands in the interactive >shell i.e. it would slow me down if I has to put in square brackets >quotes and commas every time I wanted to type a command in the shell >like ['ls', '-l', '~/mystuff']. But this convenience comes at a price >in that it becomes hard to handle strings that have spaces in them. >I've never really figured out how to write a non-trivial shell script >that can properly handle whitespace (it's easier to just avoid putting >spaces in filenames - or write the script in Python). > >So really I think it's a good thing that Python lets you clearly >delineate each string in the argument list: > subprocess.call(['ls', '-l', foldername]) >without needing to worry about whether or not foldername contains spaces. Thank you everybody for all your replies! I will use a list from now on. Also good to know that this works a little differently under Windows. Albert-Jan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
On 26 February 2014 08:50, Albert-Jan Roskam wrote: > > Yesterday evening (it was *late* so forgive me if I wrong) I realized that > part of my confusion was also caused by the fact that I ran my code in Idle. > If I called subprocess.call with a list argument, it returned code 0 > (success) but the generated sphinx code was not the same as when I ran the > program from the terminal. I concluded that this is some weird interaction > between Idle (which may also run in a subprocess??) and my own program. I > also had no problems when I ran (in the terminal): python -c "import > subprocess; subprocess.call(['sphinx-apidoc'...(etc)])" > > I was surprised that the list elements of the argument for subprocess.call > *have to be* separate tokens, e.g. (earlier in one of Danny's replies): a > token (list element) '-f -F' won't work, it has to be two separate > elements/tokens: '-f', '-F'. Apparently, subprocess.call is more than just a > convenience function that " ".joins the list and escapes the resulting string. At the operating system level there is no "command line" as such on posix (Linux, Mac etc.). The command to run a program is a string giving the name of the program and you can also pass a list of strings to the program as "arguments". The command line is something that happens in the shell (e.g. bash). Whereas Python would use quotes and commas to separate strings in a list of strings the shell just assumes that whitespace separates strings. So in Python you would write ['qwe', 'asd', 'zxc zxc'] and it's clear that you have a list of three strings. In the shell (e.g. bash) you would type qwe asd "zxc zxc" and the shell understands that as three separate strings. The quotes are needed to distinguish it from 4 strings because shell syntax assumes that spaces separate strings. So the idea of the "command line" as a single string that must be split on whitespace is just something that happens in the shell. It is supposed to be a convenient way of typing commands in the interactive shell i.e. it would slow me down if I has to put in square brackets quotes and commas every time I wanted to type a command in the shell like ['ls', '-l', '~/mystuff']. But this convenience comes at a price in that it becomes hard to handle strings that have spaces in them. I've never really figured out how to write a non-trivial shell script that can properly handle whitespace (it's easier to just avoid putting spaces in filenames - or write the script in Python). So really I think it's a good thing that Python lets you clearly delineate each string in the argument list: subprocess.call(['ls', '-l', foldername]) without needing to worry about whether or not foldername contains spaces. Oscar ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
On Wed, Feb 26, 2014 at 3:50 AM, Albert-Jan Roskam wrote: > On Tue, Feb 25, 2014 at 4:54 PM, Dave Angel wrote: >> CreateProcess has its own design bobbles as well. For >> example, if you forget to put quotes around the program >> name, it will happily try to add ".exe" to *multiple* >> places in the hopes that one of them will work. >> Adding a file c:\program.exe to a system will blow up >> lots of programs that were working by mistake for years. > > Yesterday evening (it was *late* so forgive me if I wrong) I > realized that part of my confusion was also caused by the fact > that I ran my code in Idle. If I called subprocess.call with a > list argument, it returned code 0 (success) but the generated > sphinx code was not the same as when I ran the program from > the terminal. I concluded that this is some weird interaction > between Idle (which may also run in a subprocess??) and my own > program. I also had no problems when I ran (in the terminal): > python -c "import subprocess; > subprocess.call(['sphinx-apidoc'...(etc)])" Run IDLE from the terminal to have sphinx-apidoc inherit the terminal for standard I/O. Then you can see the TTY output. I assume you're using a POSIX system. I tacked on the bit about Windows CreateProcess just so you'd be aware of the differences. I'd flip the quoting around in a POSIX shell: python -c 'import subprocess; subprocess.call(["sphinx-apidoc", "..."])'. The shell expands $ and `cmd` in a double-quoted string: $ msg=spam $ echo "$msg" spam $ echo "`uname -o`" GNU/Linux But not in a single-quoted string: $ echo '$msg' $msg $ echo '`uname -o`' `uname -o` > I was surprised that the list elements of the argument for > subprocess.call *have to be* separate tokens, e.g. (earlier in > one of Danny's replies): a token (list element) '-f -F' won't > work, it has to be two separate elements/tokens: '-f', '-F'. > Apparently, subprocess.call is more than just a convenience > function that " ".joins the list and escapes the resulting > string. subprocess.call doesn't join the arguments: def call(*popenargs, **kwargs): return Popen(*popenargs, **kwargs).wait() On a POSIX system, Popen._execute_child doesn't join the arguments, either. It sets executable=args[0] and calls os.execvp(executable, args) in the forked child process. If executable is a relative path, os.execvp uses a loop over the paths in the PATH environment variable to create an absolute path. In CPython, os.execvp calls posix.execv(path, args), which is written in C to call the system function execv(const char *path, char *const argv[]). This replaces the current process image with the executable file located at the absolute `path`. To set up the system call, posix.execv transforms the tuple/list of args into an array of char * pointers of length len(args) + 1. The final item of the array is the NULL terminator. Any unicode strings are encoded with the file-system encoding (probably UTF-8). If the new process image isn't successfully executed (replacing Python in the process), then OSError is raised. This is in the forked child, so Popen._execute_child has to pickle the exception and write it back to the parent process via os.write(errpipe_write, pickle.dumps(exc_value)). It reads the `data` from the other end of the pipe in the parent, and if it's non-empty it'll `raise pickle.loads(data)`. That's the source of the OSError from calling subprocess.call(cmd), where cmd was the entire command line as a string. As to what a program does if you mash two options into a single item of its argv array, such as '-f -F', assuming it isn't trying to be clever and expects the command-line to be parsed by a standard POSIX shell, then '-f -F' won't match any of its known options. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
Regards, Albert-Jan ~~ All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us? ~~ On Tue, 2/25/14, eryksun wrote: Subject: Re: [Tutor] subprocess.call list vs. str argument To: "Dave Angel" Cc: tutor@python.org Date: Tuesday, February 25, 2014, 11:30 PM On Tue, Feb 25, 2014 at 4:54 PM, Dave Angel wrote: > CreateProcess has its own design bobbles as well. For example, if > you forget to put quotes around the program name, it will > happily try to add ".exe" to *multiple* places in the hopes that > one of them will work. > > Adding a file c:\program.exe to a system will blow up lots of > programs that were working by mistake for years. Yesterday evening (it was *late* so forgive me if I wrong) I realized that part of my confusion was also caused by the fact that I ran my code in Idle. If I called subprocess.call with a list argument, it returned code 0 (success) but the generated sphinx code was not the same as when I ran the program from the terminal. I concluded that this is some weird interaction between Idle (which may also run in a subprocess??) and my own program. I also had no problems when I ran (in the terminal): python -c "import subprocess; subprocess.call(['sphinx-apidoc'...(etc)])" I was surprised that the list elements of the argument for subprocess.call *have to be* separate tokens, e.g. (earlier in one of Danny's replies): a token (list element) '-f -F' won't work, it has to be two separate elements/tokens: '-f', '-F'. Apparently, subprocess.call is more than just a convenience function that " ".joins the list and escapes the resulting string. regards, Albert-Jan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
On Tue, Feb 25, 2014 at 4:54 PM, Dave Angel wrote: > CreateProcess has its own design bobbles as well. For example, if > you forget to put quotes around the program name, it will > happily try to add ".exe" to *multiple* places in the hopes that > one of them will work. > > Adding a file c:\program.exe to a system will blow up lots of > programs that were working by mistake for years. Yes, using a string requires quoting the path for the executable if you aren't using lpApplicationName (i.e. Popen's "executable"). I noticed for shell=True they aren't using list2cmdline for COMPSPEC; they just trust it. So I wrote the following test to exemplify the problem you're talking about: >>> import os, subprocess >>> open('temp.c', 'w').write(r''' ... #include ... int main(int argc, char *argv[]) { ... printf("\n\n*** spam! ***\n\n"); ... return 0; ... }''') 107 >>> os.system('cl temp.c /FeC:\\Program.exe 1>nul 2>&1') 0 >>> os.environ['COMSPEC'] = r'C:\Program Files\doesnt\matter' >>> p = subprocess.Popen('dir', shell=True) >>> *** spam! *** ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
eryksun Wrote in message: > > > FYI, in Windows the situation is different. CreateProcess takes a > string argument, so the setup code changes to the following: > > > It's fine to use a string for args in Windows, and you may need to if > the program parses the command line differently than list2cmdline, > which escapes the arguments according to the rules used by Microsoft's > CRT. CreateProcess has its own design bobbles as well. For example, if you forget to put quotes around the program name, it will happily try to add ".exe" to *multiple* places in the hopes that one of them will work. Adding a file c:\program.exe to a system will blow up lots of programs that were working by mistake for years. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
See the comment about 'args' in: http://docs.python.org/2/library/subprocess.html#frequently-used-arguments which says: """args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names). If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.""" This is an unfortunate pain-point in the subprocess API. Personally, I think that having subprocess.call optionally accept a single string is a wrong API decision because it causes so much confusion. You can tell there's a schizophrenia in that part of the documentation: it says something to the effect of, "If you want to use a single string, you have to use shell=True". And a few paragraphs later, it says, "But don't use shell=True!" I would strongly suggest not trying to use the single string interface to subprocess: it's too easy to get wrong. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
On Tue, Feb 25, 2014 at 2:52 PM, Albert-Jan Roskam wrote: > Here is why I used "shell=True" before. Is it related to the > file paths? > > cmd = (r'sphinx-apidoc ' >r'-f -F ' >r'-H "%(title)s" ' >r'-A "%(author)s" ' >r'-V "%(version)s" ' >r'-o %(output_dir)s %(input_dir)s') % locals() > retcode = subprocess.call(cmd) Take a look at the code in subprocess._execute_child for POSIX platforms: if isinstance(args, types.StringTypes): args = [args] else: args = list(args) if shell: args = ["/bin/sh", "-c"] + args if executable: args[0] = executable if executable is None: executable = args[0] It calls os.execvp in the child process, for which you want the executable file argument to be 'sphinx-apidoc'. But without shell=True, you can see that Popen uses your entire cmd string as the executable. FYI, in Windows the situation is different. CreateProcess takes a string argument, so the setup code changes to the following: if not isinstance(args, types.StringTypes): args = list2cmdline(args) # Process startup details if startupinfo is None: startupinfo = STARTUPINFO() if None not in (p2cread, c2pwrite, errwrite): startupinfo.dwFlags |= _subprocess.STARTF_USESTDHANDLES startupinfo.hStdInput = p2cread startupinfo.hStdOutput = c2pwrite startupinfo.hStdError = errwrite if shell: startupinfo.dwFlags |= _subprocess.STARTF_USESHOWWINDOW startupinfo.wShowWindow = _subprocess.SW_HIDE comspec = os.environ.get("COMSPEC", "cmd.exe") args = '{} /c "{}"'.format (comspec, args) It's fine to use a string for args in Windows, and you may need to if the program parses the command line differently than list2cmdline, which escapes the arguments according to the rules used by Microsoft's CRT. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
Albert-Jan Roskam Wrote in message: > > > - Original Message - > >> From: Danny Yoo >> To: Peter Otten <__pete...@web.de> >> Cc: Python Tutor Mailing List >> Sent: Monday, February 24, 2014 11:01 PM >> Subject: Re: [Tutor] subprocess.call list vs. str argument >> >> Last comment (apologies for being so fragmented!). I don't know why >> the literal strings there are raw there. Altogether, I'd expect: >> >> # >> cmd1 = ['sphinx-apidoc', >>'-f', >>'-F', >>'-H', title, >>'-A', author, >>'-V', version, >>'-o', output_dir, >>input_dir] >> retcode = subprocess.call(cmd1, shell=False) >> >> # > > > Hi Danny, Peter, > > Thanks for helping me. Danny, your example works indeed and it is also very > readable. And now I do not need to use shell=True anymore. The raw strings > were a leftover from earlier attempts that contained the entire path (not > even relevant under Linux, but I still did it). Here is why I used > "shell=True" before. Is it related to the file paths? > > cmd = (r'sphinx-apidoc ' >r'-f -F ' >r'-H "%(title)s" ' >r'-A "%(author)s" ' >r'-V "%(version)s" ' >r'-o %(output_dir)s %(input_dir)s') % locals() > retcode = subprocess.call(cmd) > Traceback (most recent call last): > File "/home/albertjan/Downloads/documenter.py", line 188, in > rst_to_html(input_dir, output_dir, title="Test", > author=getpass.getuser(), version="1.0.0") > File "/home/albertjan/Downloads/documenter.py", line 118, in rst_to_html > retcode = subprocess.call(cmd) > File "/usr/lib/python2.7/subprocess.py", line 493, in call > return Popen(*popenargs, **kwargs).wait() > File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ > errread, errwrite) > File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child > raise child_exception > OSError: [Errno 2] No such file or directory > Here you are feeding a string to the call function, while Danny was using a list. With a string, you must use shell=True, so the shell can turn it into a list. Usually much easier to use a list in the first place. -- DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
- Original Message - > From: Danny Yoo > To: Peter Otten <__pete...@web.de> > Cc: Python Tutor Mailing List > Sent: Monday, February 24, 2014 11:01 PM > Subject: Re: [Tutor] subprocess.call list vs. str argument > > Last comment (apologies for being so fragmented!). I don't know why > the literal strings there are raw there. Altogether, I'd expect: > > # > cmd1 = ['sphinx-apidoc', > '-f', > '-F', > '-H', title, > '-A', author, > '-V', version, > '-o', output_dir, > input_dir] > retcode = subprocess.call(cmd1, shell=False) > > # Hi Danny, Peter, Thanks for helping me. Danny, your example works indeed and it is also very readable. And now I do not need to use shell=True anymore. The raw strings were a leftover from earlier attempts that contained the entire path (not even relevant under Linux, but I still did it). Here is why I used "shell=True" before. Is it related to the file paths? cmd = (r'sphinx-apidoc ' r'-f -F ' r'-H "%(title)s" ' r'-A "%(author)s" ' r'-V "%(version)s" ' r'-o %(output_dir)s %(input_dir)s') % locals() retcode = subprocess.call(cmd) Traceback (most recent call last): File "/home/albertjan/Downloads/documenter.py", line 188, in rst_to_html(input_dir, output_dir, title="Test", author=getpass.getuser(), version="1.0.0") File "/home/albertjan/Downloads/documenter.py", line 118, in rst_to_html retcode = subprocess.call(cmd) File "/usr/lib/python2.7/subprocess.py", line 493, in call return Popen(*popenargs, **kwargs).wait() File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory regards, Albert-Jan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
Last comment (apologies for being so fragmented!). I don't know why the literal strings there are raw there. Altogether, I'd expect: # cmd1 = ['sphinx-apidoc', '-f', '-F', '-H', title, '-A', author, '-V', version, '-o', output_dir, input_dir] retcode = subprocess.call(cmd1, shell=False) # ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
There are a few issues there. I'd also recommend not trying to shell-quote these manually, # in the argument list of os.subprocess: r'-H', '"%s"' % title, r'-A', '"%s"' % author, r'-V', '"%s"' % version, Rather, just do the simpler thing: r'-H', title, r'-A', author, r'-V', version, in conjunction with passing the "shell=False" keyword argument. Don't escape. Just pass the arguments as is. As far as I can tell, trying to do shell escaping is not only unnecessary here, but doing without it makes the code cleaner safer. Maybe there's another reason why Albert-Jan's situation is different enough that "shell=True" is necessary, but the default situation should be to avoid it. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
Peter Otten wrote: >> r'-f -F', >> r'-H', '"%s"' % title, > > "title" becomes \"title\", i. e. Python puts in an extra effort to have > the quotes survive the subsequent parsing process of the shell: > print subprocess.list2cmdline(['"title"']) > \"title\" Forget that :( Danny spotted the real problem. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
Albert-Jan Roskam wrote: > Hi, > > In the code below, cmd1 and cmd2 are equivalent, as in: " ".join(cmd1) == > cmd2. But the first example returns a code 2, whereas the second runs > successfully. What is the difference? I prefer using a list as it looks a > little cleaner. Btw, shell=True is needed here. > > > # Python 2.7.3 (default, Jan 2 2013, 13:56:14) [GCC 4.7.2] on linux2 > > import subprocess > > #1# returns retcode 2 (whatever error that maybe) > title, author, version = "title", "author", "1.0.0" > output_dir, input_dir = '/tmp/input', '/tmp/output' > cmd1 = [r'sphinx-apidoc', > r'-f -F', > r'-H', '"%s"' % title, "title" becomes \"title\", i. e. Python puts in an extra effort to have the quotes survive the subsequent parsing process of the shell: >>> print subprocess.list2cmdline(['"title"']) \"title\" > r'-A', '"%s"' % author, > r'-V', '"%s"' % version, > r'-o', output_dir, input_dir] > retcode = subprocess.call(cmd1, shell=True) > assert not retcode, retcode > > #2# returns retcode 0 (succes) > cmd2 = (r'sphinx-apidoc ' > r'-f -F ' > r'-H "%(title)s" ' > r'-A "%(author)s" ' > r'-V "%(version)s" ' > r'-o %(output_dir)s %(input_dir)s') % locals() > > retcode = subprocess.call(cmd2, shell=True) > assert not retcode, retcode > > # no AssertionError > assert " ".join(cmd1) == cmd2 > > > > Thanks in advance! > > Regards, > > Albert-Jan > > > > > ~~ > > All right, but apart from the sanitation, the medicine, education, wine, > public order, irrigation, roads, a > > fresh water system, and public health, what have the Romans ever done for > us? > > ~~ > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] subprocess.call list vs. str argument
> cmd1 = [r'sphinx-apidoc', >r'-f -F', This part looks suspicious. Are you sure you don't mean: '-f', '-F' here? They need to be separate arguments. Also, you mention: > Btw, shell=True is needed here. Why do you need shell expansion on the arguments? This can be dangerous unless you know what you're doing. See all the red-backgrounded warnings the subprocess documentation. For example: http://docs.python.org/2/library/subprocess.html#frequently-used-arguments ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor