[issue13238] Add shell command helpers to shutil module

Éric Araujo added the comment: I’m not sure my question was well phrased. If I have these files: spam.py ham.py foo bar.py will a pattern of '*.py' match all of them with your functions, even the one with an embedded space? -- ___ Python tra

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: Yeah, I was thinking about this a bit more and realised that I'd rejected the "quote everything by default" approach before I had the idea of providing a custom conversion specifier to disable the implicit string conversion and quoting. So perhaps a better alt

[issue13238] Add shell command helpers to shutil module

Antoine Pitrou added the comment: > With the default whitespace escaping (which allows spaces in > filenames), wildcard matching still works (thus the list of > directories matching the "../py*" pattern), but with full quoting it > breaks (thus the "nothing named '../py*'" result). My question

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: The first version I wrote *did* automatically invoke shlex.quote on all interpolated values, but that breaks wildcard handling. You can see that in the examples I posted above. With the default whitespace escaping (which allows spaces in filenames), wildcard ma

[issue13238] Add shell command helpers to shutil module

Antoine Pitrou added the comment: > default - str + shutil.quote_ascii_whitespace > !q - str + shlex.quote > !u - unquoted (i.e. no conversion, str.format default behaviour) The default doesn't look very understandable to me. Why would you quote only some characters and not all of them? --

[issue13238] Add shell command helpers to shutil module

Éric Araujo added the comment: The custom formatter idea sounds brilliant. Can you test that auto-escaping of spaces works well with glob patterns? -- ___ Python tracker ___ _

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: Some examples: >>> import shutil >>> shutil.shell_call("du -hs {}", "../py*") 594M../py3k 579M../py3k_pristine 480M../python27 301M../python31 382M../python32 288K../python_swallowed_whole 0 >>> shutil.shell_call("du -hs {!q}", "../py*") d

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: I realised I could use the convert_field() option in the custom formatter to choose between several interpolation quoting options: default - str + shutil.quote_ascii_whitespace !q - str + shlex.quote !u - unquoted (i.e. no conversion, str.format default be

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: After a bit of thought, I realised I could use the string.Formatter API to implement a custom formatter for the shell command helpers that auto-escapes whitespace while leaving the other shell metacharacters alone (so you can still interpolate paths containing

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: I discovered a couple of APIs that were moved from the commands module to the subprocess module in 3.0: http://docs.python.org/dev/library/subprocess#subprocess.getstatusoutput However, they have issues, especially on Windows: http://bugs.python.org/issue10197

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: Considering this further, I've realised that the idea of implicit quoting for this style of helper function is misguided on another level - the parameters to be interpolated may not even be strings yet, so attempting to quote them would fail: >>> subprocess.ca

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: Unfortunately, I don't think including implicit shlex.quote() calls is going to have the effect I was originally looking for: >>> subprocess.call("du -hs ../py*", shell=True) 593M../py3k 577M../py3k_pristine 479M../python27 300M../python31 381M

[issue13238] Add shell command helpers to shutil module

Éric Araujo added the comment: s/stings/strings/ -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue13238] Add shell command helpers to shutil module

Éric Araujo added the comment: > [snip rationale about why shutil and not subprocess] I’m convinced (with one nit: sh in the shutil name does not ring a security alarm for me, as I understand it as “shell-like conveniences in nice, dont-do-nasty-things-with-stings Python” :) but the shell in c

[issue13238] Add shell command helpers to shutil module

Antoine Pitrou added the comment: > That's a fair point, but I think it actually *improves* the argument > for better helper functions Agreed :) -- ___ Python tracker ___ _

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: That's a fair point, but I think it actually *improves* the argument for better helper functions, since we can have them automatically invoke shlex.quote() on all of the arguments: def _shell_format(cmd, args, kwds): args = map(shlex.quote, args)

[issue13238] Add shell command helpers to shutil module

Antoine Pitrou added the comment: > Yes, you have to be careful that "dirname" is legal in the shell, but > that usually isn't a big problem in practice, because dirname came > from a previous listdir call, or you otherwise know that it's valid to > interpolate it into the command I don't under

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: It's a flow thing. This idea was kicked off by the process of translating a large Perl script to Python and paying attention to what the translation made *worse*. One of the big things it made worse was the translation of "qx" (quoted executable) strings. In P

[issue13238] Add shell command helpers to shutil module

Antoine Pitrou added the comment: Hum, in: return_code = shellcmd.shell_call('ls -l {}', dirname) listing = shellcmd.check_shell_output('ls -l {}', dirname) ...how do you know that dirname doesn't need some kind of escaping? This is not only a security issue, but a bug. Even if security doesn'

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: Initially, because I was suggesting the names shadow the subprocess convenience functions so they *had* to live in a different namespace. However, even after changing the names to explicitly include "shell", I'd like to keep them away from the general subproces

[issue13238] Add shell command helpers to shutil module

Éric Araujo added the comment: Why not keeping these helpers in subprocess? -- nosy: +eric.araujo ___ Python tracker ___ ___ Python-b

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: Of the 3 available options (mod style, string.Template and str.format), yes, str.format is the best choice. If people want the shell meaning of the braces, they can escape them by doubling them up in the command string. --

[issue13238] Add shell command helpers to shutil module

Chris Rebert added the comment: Is format() really the best choice here, considering that {}s already have a meaning in the shell? -- nosy: +cvrebert ___ Python tracker ___ ___

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: Perhaps a better idea would be to use different names, so it's clearer at the point of invocation that the shell is being invoked (and hence shell injection attacks are a potential concern). For example: shell_call check_shell_call check_shell_output Tha

[issue13238] Add shell command helpers to shutil module

Nick Coghlan added the comment: And that's exactly the problem - a web developer's or security auditor's "shell injection" is a system administrator's "this language sucks". These wrappers are the kind of thing you want for shell invocations when using Python as a replacement for a shell scri

[issue13238] Add shell command helpers to shutil module

Alex Gaynor added the comment: These feel like a shell injection waiting to happen to me. -- nosy: +alex ___ Python tracker ___ ___ P

[issue13238] Add shell command helpers to shutil module

New submission from Nick Coghlan : I've been doing a few systems administration tasks with Python recently, and shell command invocation directly via the subprocess module is annoyingly clunky (even with the new convenience APIs). Since subprocess needs to avoid the shell by default for securi