[issue1521950] shlex.split() does not tokenize like the shell

Roundup Robot added the comment: New changeset ea99e2f0b829 by Vinay Sajip in branch 'default': Closes #1521950: Made shlex parsing more shell-like. https://hg.python.org/cpython/rev/ea99e2f0b829 -- nosy: +python-dev resolution: -> fixed stage: patch review -> resolved status: open ->

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : -- assignee: -> vinay.sajip ___ Python tracker ___ ___

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: Okay, I've updated with a new patch addressing SilentGhost's comments, and addressed the comments on that patch. If I don't hear any objections by Friday, I plan to commit this change. -- ___ Python tracker

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip : Added file: http://bugs.python.org/file43831/refresh-2016.diff ___ Python tracker ___

[issue1521950] shlex.split() does not tokenize like the shell

R. David Murray added the comment: No objection from me. I'm not likely to have the time to give it the kind of thorough review I'd *like* to, but I don't think it is really needed. -- ___ Python tracker

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: This has been knocking around since 3.3, but never got enough attention to make it in. Barring objections from anyone, I'd like to commit this patch once I check that it applies cleanly against 3.6, before we get into 3.6 beta. -- versions: +Python 3.6

[issue1521950] shlex.split() does not tokenize like the shell

Andrey Kislyuk added the comment: Is there any chance of getting this into 3.6? We are still in a situation where the shlex module misleads developers into believing that it has functionality to parse things the way the shell does. I've had to vendor the copy of shlex with patches from this

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip vinay_sa...@yahoo.co.uk: Added file: http://bugs.python.org/file36772/80eea6bd898c.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Chris Rebert pyb...@rebertia.com: -- nosy: +cvrebert ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___ ___ Python-bugs-list

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip added the comment: Let's hope we can get this into 3.5. I updated my patch a while ago to address RDM's comments. -- versions: +Python 3.5 -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip vinay_sa...@yahoo.co.uk: -- versions: +Python 3.4 -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___ ___

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip vinay_sa...@yahoo.co.uk: Added file: http://bugs.python.org/file25809/388411be9b61.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: I've updated the patch following comments by RDM - it probably could do with a code review (now that I've addressed RDM's comments on the docs). -- ___ Python tracker rep...@bugs.python.org

[issue1521950] shlex.split() does not tokenize like the shell

R. David Murray rdmur...@bitdance.com added the comment: Review, including a code-but-not-algorithm review :), posted. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___ ___

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip vinay_sa...@yahoo.co.uk: Added file: http://bugs.python.org/file25365/9252961a03e7.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo mer...@netwok.org added the comment: I'd like to take a look at this (I wasn't aware of it before). Are you interested in shlex in general or only this bug? If the former, then I’ll try to remember to make you nosy on future issues. BTW, what is the shlex unicode bug you

[issue1521950] shlex.split() does not tokenize like the shell

R. David Murray rdmur...@bitdance.com added the comment: I am interested in shell stuff in general. The unicode bug is issue 1170. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: I believe Dan meant that the behaviour of shlex.split() now is different from what it was when he first raised the issue (in July 2006). Looking at the default branch of CPython, this is what I see: Python 3.3.0a2+ (default:ff6593aa8376,

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: I've received no comments on the latest revision of my patch (incorporating comments on the previous version); is it OK to commit this? -- ___ Python tracker rep...@bugs.python.org

[issue1521950] shlex.split() does not tokenize like the shell

R. David Murray rdmur...@bitdance.com added the comment: I'd like to take a look at this (I wasn't aware of it before). I'll try to do that some time in the next 24 hours, and if I don't you shouldn't wait for me :) Did you address Dan's concern about 'old' possibly not matching the old

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: Éric Araujo mer...@netwok.org added the comment: I did not fully get what you meant here, but the example you added to the doc made it clear.  Is this covered by tests? Yes, I believe that testSyntaxSplitCustom covers this. Overall great

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian robo...@users.sourceforge.net added the comment: I haven't been following this much. Sorry. My day job isn't in this area any more (and I'm stuck using 2.4 :-(). Looking at the docs, I notice the old is different from what it used to be. Notably: 'e;' gets split into two

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo mer...@netwok.org added the comment: Previously when punctuation chars were set, wordchars was being augmented by '-'. This was incomplete, so the augmentation is now with '~-./*?=' which allows for wildcards, filename chars and argument flags. I did not fully get what you

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Gustavo Niemeyer gust...@niemeyer.net: -- nosy: -niemeyer ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___ ___

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip vinay_sa...@yahoo.co.uk: Added file: http://bugs.python.org/file24590/079ab75d29a4.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: I updated the patch to reflect Éric's comments on Rietveld, but there are also some other changes: Previously when punctuation chars were set, wordchars was being augmented by '-'. This was incomplete, so the augmentation is now with

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: Plus I also changed a few instances of the anachronism a = a + b to a += b -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo mer...@netwok.org added the comment: This time you should have received an email from Rietveld, I made sure that your ID was expanded to an email address. I like all the suggestions you made in reply to my comments. -- ___ Python

[issue1521950] shlex.split() does not tokenize like the shell

Vinay Sajip vinay_sa...@yahoo.co.uk added the comment: I've made a patch which implements this functionality, together with docs and tests. Please review. -- hgrepos: +99 nosy: +vinay.sajip stage: test needed - patch review ___ Python tracker

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Vinay Sajip vinay_sa...@yahoo.co.uk: Added file: http://bugs.python.org/file24158/9e12275eec25.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Éric Araujo mer...@netwok.org: Removed file: http://bugs.python.org/file23778/ref_shlex.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Éric Araujo mer...@netwok.org: Removed file: http://bugs.python.org/file23779/test_shlex.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo mer...@netwok.org added the comment: Thanks for the diff and test. (I removed the older versions; there are “edit” links in the list of files leading to pages where it’s possible to remove them, if one has the required permissions.) Your script passes with dash, which is probably

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian robo...@users.sourceforge.net added the comment: On Sat, Nov 26, 2011 at 7:12 AM, Éric Araujo rep...@bugs.python.org wrote: Your script passes with dash, which is probably the most POSIX-compliant shell we can find.  (bash has extensions, zsh/csh don’t use the POSIX shell

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo mer...@netwok.org added the comment: I was just looking for a reference where I didn't have to sift through tons of documentation. Sure :) That’s why I suggest using dash for quick tests and rely on the work of other people who did read the POSIX spec. I’ll have to check it too

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian robo...@users.sourceforge.net added the comment: Sure :)  That’s why I suggest using dash for quick tests and rely on the work of other people who did read the POSIX spec.  I’ll have to check it too before committing a patch. The point of ref_shlex.py is that all shells act

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo mer...@netwok.org added the comment: Of course, that's how it's used. That's all it can do right now. :) What I meant is that it is *meant* to be used in this way. I was was splitting and combining commands (using ;, , and ||) and then running the resulting (mega) one liners

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian robo...@users.sourceforge.net added the comment: I've attached a diff to test_shlex.py and a script that I used to verify what the shells actually do. Both are relative to Python-3.2.2/Lib/test I'm completely ignoring the quotes issue for now. That should probably be an

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian robo...@users.sourceforge.net added the comment: I just realized that I left out a major case. The shell will also split (). I think this is now complete. If you do man bash and skip down to DEFINITONS it lists all the control characters. I've attached updated versions of

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo mer...@netwok.org added the comment: Thanks for the comments. There are really two cases in one bug. The first part is that the shell will split tokens at characters that shlex doesn't. The handling of , |, ;, , and could be done by adjusting the definition of

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian robo...@users.sourceforge.net added the comment: Of course, that's how it's used. That's all it can do right now. I was was splitting and combining commands (using ;, , and ||) and then running the resulting (mega) one liners over ssh. It still gets run by a shell, but I was

[issue1521950] shlex.split() does not tokenize like the shell

Éric Araujo mer...@netwok.org added the comment: Thanks for the report. Would you like to work on a patch, or translate your examples into unit tests? The docs do not mention “” at all, and platform discrepancies have to be taken into account too, so I really don’t know if this is a bug fix

[issue1521950] shlex.split() does not tokenize like the shell

Dan Christian robo...@users.sourceforge.net added the comment: It's been a while since I looked at this.  I'm not really in a position to contribute code/tests right now; but I can comment. I don't think POSIX mode existed when I first reported this, but that's where it makes sense.  I think

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Mark Lawrence breamore...@yahoo.co.uk: -- versions: +Python 3.2 -Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___

[issue1521950] shlex.split() does not tokenize like the shell

Changes by Daniel Diniz aja...@gmail.com: -- stage: - test needed type: - feature request versions: +Python 2.7 -Python 2.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1521950 ___