Re: line to argv transformation
On 16-06-14 13:09, Chris Angelico wrote: > On Mon, Jun 16, 2014 at 8:51 PM, Tim Chase > wrote: >> On 2014-06-16 20:41, Chris Angelico wrote: >>> Oops! I made the cardinal error of trying in one and assuming it'd >>> work in both. Just needs a b prefix on the split string: >>> >>> def shell_split(cmd): >>> return subprocess.check_output("""python -c 'import sys; >>> print("\\0".join(sys.argv[1:]))' >>> """+cmd,shell=True)[:-1].split(b"\0") >>> >>> You'll get back a list of byte strings, in any case. Feel free to >>> pass them through a decode operation, or to incorporate a .decode() >>> into the above stream, as you wish. >> Tested on Win32? The behavior of "shell expansion" on Windows >> cmd.exe differs from most *nix shells (i.e., it *doesn't* expand, so >> you have to do it yourself), so Antoon would need to describe the >> desired behavior on Win32. > Well, his example commands began "ls", which is a common Unix command, > but isn't present on most Windows systems. If he'd started out with > something that looked more Windowsish, I'd totally understand - you > start with a single line, and you need to do what the cmd.exe shell > hasn't done for you. (Although splitting is done, so it still wouldn't > be quite as clear.) But he said "treated as a command line". So that's > exactly what I did. :) He didn't ask about globbing, he asked about > doing what the shell does... maybe he wants variable expansion too? That would be interresting too. The problem with your solution would be that it would only substitute environment variables, and not "shell variables" within the interactive session. Of course I could use os.putenv to put all those variables in the environment but that might have troublesome effects on other subprocesses. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On 2014-06-16 13:51, Antoon Pardon wrote: > >>> shlex.split("ls *.py") > ['ls', '*.py'] > >>> shlex.split("ls '*.py'") > ['ls', '*.py'] To accommodate this, I'd probably just clone the shlib.py to my local project under a new name and then tweak the source to emit whether a token was quoted or not, something like the diff below. You can then iterate over your string/token-stream and know whether it was quoted or not, allowing you to do any post-processing/globbing on that file. -tkc --- /usr/lib/python2.7/shlex.py 2014-03-13 05:54:53.0 -0500 +++ /home/tim/tmp/myshlex.py2014-06-16 07:39:34.130645270 -0500 @@ -93,29 +93,30 @@ print "shlex: popping token " + repr(tok) return tok # No pushback. Get a token. -raw = self.read_token() +was_quoted, raw = self.read_token() # Handle inclusions if self.source is not None: while raw == self.source: -spec = self.sourcehook(self.read_token()) +was_quoted, token = self.read_token() +spec = self.sourcehook(roken) if spec: (newfile, newstream) = spec self.push_source(newstream, newfile) -raw = self.get_token() +was_quoted, raw = self.get_token() # Maybe we got EOF instead? while raw == self.eof: if not self.filestack: return self.eof else: self.pop_source() -raw = self.get_token() +was_quoted, raw = self.get_token() # Neither inclusion nor EOF if self.debug >= 1: if raw != self.eof: print "shlex: token=" + repr(raw) else: print "shlex: token=EOF" -return raw +return was_quoted, raw def read_token(self): quoted = False @@ -243,7 +244,7 @@ print "shlex: raw token=" + repr(result) else: print "shlex: raw token=EOF" -return result +return quoted, result def sourcehook(self, newfile): "Hook called on a filename to be sourced." @@ -266,10 +267,10 @@ return self def next(self): -token = self.get_token() +was_quoted, token = self.get_token() if token == self.eof: raise StopIteration -return token +return was_quoted, token def split(s, comments=False, posix=True): lex = shlex(s, posix=posix) @@ -285,7 +286,7 @@ file = sys.argv[1] lexer = shlex(open(file), file) while 1: -tt = lexer.get_token() +was_quoted, tt = lexer.get_token() if tt: print "Token: " + repr(tt) else: -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On 16-06-14 13:01, Peter Otten wrote: > Antoon Pardon wrote: > >> I am looking for an interface that takes a string as argument. The >> string is to be treated as if it is a command line and transformed into >> an argv list. >> >> "ls file" -> ['ls', 'file'] >> "ls *.py" -> ['ls', 'file1.py', 'file2.py', ...] >> "ls '*.py'" -> ['ls', '*.py'] >> >> Does something like this already exist? I looked around but seem to find >> only things only partially do things like this, like shlex.split. > You might combine shlex and glob: > > def parse_and_expand(s): > parts = shlex.split(s) > expanded = [] > for part in parts: > matches = glob.glob(part) > if matches: > expanded.extend(sorted(matches)) > else: > expanded.append(part) > return expanded No that doesn't work because of this: >>> shlex.split("ls *.py") ['ls', '*.py'] >>> shlex.split("ls '*.py'") ['ls', '*.py'] >>> After the split you have no idea wether the glob pattern you see was quoted or not. -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On Mon, Jun 16, 2014 at 8:59 PM, Marko Rauhamaa wrote: > Chris Angelico : > >> def shell_split(cmd): >> return subprocess.check_output("""python -c 'import sys; >> print("\\0".join(sys.argv[1:]))' """+cmd,shell=True)[:-1].split(b"\0") >> >> You'll get back a list of byte strings, in any case. Feel free to pass >> them through a decode operation, or to incorporate a .decode() into >> the above stream, as you wish. > > Now you are subject to the quirks of /bin/sh. For example, '*.xyz' > expands itself ('*.xyz') if there is no match in the current working > directory. > > Moreover, you need to guard against arguments like > >$(cat ~/.ssh/id_dsa) > > which would spill the beans. As I said in response to Tim, the somewhat underspecified question does leave open followups of whether both of those would be features, rather than bugs. For instance, if "*.py" should expand to a list of all files matching that glob, should "[123].py" expand to any files matching that pattern? I'm not sure that your typical glob function handles that. And should "spam{eggs,spam}" become "spameggs","spamspam"? (Though that one's bash-specific, I believe.) Where do you draw the line? My reading is that it should be one of two options: 1) Split, according to shell rules, and then glob. Nothing more. No square brackets (probably), no braces, nothing. 2) Do exactly what $CHELL would do, for some value of $CHELL. But neither is quite clear, and I can see exactly why there isn't anything in the stdlib. And what shell do you want to imitate, for option 2? Using /bin/sh makes a lot of sense... but so does /bin/bash. Or maybe you should use the user's own login shell, if you're wrapping a command prompt in some way. Perhaps you want to use GLaDOS; but remember that although fun and learning are the primary goals of this activity, serious injuries may occur, especially when using backticks or $( ) in the command. OP needs to specify better. Otherwise Black Mesa will get the contract. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On 2014-06-16 20:41, Chris Angelico wrote: > Oops! I made the cardinal error of trying in one and assuming it'd > work in both. Just needs a b prefix on the split string: > > def shell_split(cmd): > return subprocess.check_output("""python -c 'import sys; > print("\\0".join(sys.argv[1:]))' > """+cmd,shell=True)[:-1].split(b"\0") > > You'll get back a list of byte strings, in any case. Feel free to > pass them through a decode operation, or to incorporate a .decode() > into the above stream, as you wish. Tested on Win32? The behavior of "shell expansion" on Windows cmd.exe differs from most *nix shells (i.e., it *doesn't* expand, so you have to do it yourself), so Antoon would need to describe the desired behavior on Win32. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On Mon, Jun 16, 2014 at 8:51 PM, Tim Chase wrote: > On 2014-06-16 20:41, Chris Angelico wrote: >> Oops! I made the cardinal error of trying in one and assuming it'd >> work in both. Just needs a b prefix on the split string: >> >> def shell_split(cmd): >> return subprocess.check_output("""python -c 'import sys; >> print("\\0".join(sys.argv[1:]))' >> """+cmd,shell=True)[:-1].split(b"\0") >> >> You'll get back a list of byte strings, in any case. Feel free to >> pass them through a decode operation, or to incorporate a .decode() >> into the above stream, as you wish. > > Tested on Win32? The behavior of "shell expansion" on Windows > cmd.exe differs from most *nix shells (i.e., it *doesn't* expand, so > you have to do it yourself), so Antoon would need to describe the > desired behavior on Win32. Well, his example commands began "ls", which is a common Unix command, but isn't present on most Windows systems. If he'd started out with something that looked more Windowsish, I'd totally understand - you start with a single line, and you need to do what the cmd.exe shell hasn't done for you. (Although splitting is done, so it still wouldn't be quite as clear.) But he said "treated as a command line". So that's exactly what I did. :) He didn't ask about globbing, he asked about doing what the shell does... maybe he wants variable expansion too? >>> shell_split("echo $DISPLAY") [b'echo', b'localhost:10.0'] (This is running over SSH, so my $DISPLAY is a perhaps unusual value.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
Antoon Pardon wrote: > I am looking for an interface that takes a string as argument. The > string is to be treated as if it is a command line and transformed into > an argv list. > > "ls file" -> ['ls', 'file'] > "ls *.py" -> ['ls', 'file1.py', 'file2.py', ...] > "ls '*.py'" -> ['ls', '*.py'] > > Does something like this already exist? I looked around but seem to find > only things only partially do things like this, like shlex.split. You might combine shlex and glob: def parse_and_expand(s): parts = shlex.split(s) expanded = [] for part in parts: matches = glob.glob(part) if matches: expanded.extend(sorted(matches)) else: expanded.append(part) return expanded -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
Chris Angelico : > def shell_split(cmd): > return subprocess.check_output("""python -c 'import sys; > print("\\0".join(sys.argv[1:]))' """+cmd,shell=True)[:-1].split(b"\0") > > You'll get back a list of byte strings, in any case. Feel free to pass > them through a decode operation, or to incorporate a .decode() into > the above stream, as you wish. Now you are subject to the quirks of /bin/sh. For example, '*.xyz' expands itself ('*.xyz') if there is no match in the current working directory. Moreover, you need to guard against arguments like $(cat ~/.ssh/id_dsa) which would spill the beans. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On Mon, Jun 16, 2014 at 8:24 PM, Antoon Pardon wrote: > On 16-06-14 12:06, Chris Angelico wrote: > >> def shell_split(cmd): >> return subprocess.check_output("""python -c 'import sys; >> print("\\0".join(sys.argv[1:]))' """+cmd,shell=True)[:-1].split("\0") > > Nice idea, unfortunatly it doesn't work in python3.3 > shell_split("ls *.py") > Traceback (most recent call last): > File "", line 1, in > File "", line 3, in shell_split > TypeError: Type str doesn't support the buffer API Oops! I made the cardinal error of trying in one and assuming it'd work in both. Just needs a b prefix on the split string: def shell_split(cmd): return subprocess.check_output("""python -c 'import sys; print("\\0".join(sys.argv[1:]))' """+cmd,shell=True)[:-1].split(b"\0") You'll get back a list of byte strings, in any case. Feel free to pass them through a decode operation, or to incorporate a .decode() into the above stream, as you wish. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On 16-06-14 12:06, Chris Angelico wrote: > def shell_split(cmd): > return subprocess.check_output("""python -c 'import sys; > print("\\0".join(sys.argv[1:]))' """+cmd,shell=True)[:-1].split("\0") Nice idea, unfortunatly it doesn't work in python3.3 >>> shell_split("ls *.py") Traceback (most recent call last): File "", line 1, in File "", line 3, in shell_split TypeError: Type str doesn't support the buffer API >>> -- https://mail.python.org/mailman/listinfo/python-list
Re: line to argv transformation
On Mon, Jun 16, 2014 at 7:56 PM, Antoon Pardon wrote: > I am looking for an interface that takes a string as argument. The > string is to be treated as if it is a command line and transformed into > an argv list. > > "ls file" -> ['ls', 'file'] > "ls *.py" -> ['ls', 'file1.py', 'file2.py', ...] > "ls '*.py'" -> ['ls', '*.py'] > > Does something like this already exist? I looked around but seem to find > only things only partially do things like this, like shlex.split. def shell_split(cmd): return subprocess.check_output("""python -c 'import sys; print("\\0".join(sys.argv[1:]))' """+cmd,shell=True)[:-1].split("\0") :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
line to argv transformation
I am looking for an interface that takes a string as argument. The string is to be treated as if it is a command line and transformed into an argv list. "ls file" -> ['ls', 'file'] "ls *.py" -> ['ls', 'file1.py', 'file2.py', ...] "ls '*.py'" -> ['ls', '*.py'] Does something like this already exist? I looked around but seem to find only things only partially do things like this, like shlex.split. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list