[issue2134] function generate_tokens at tokenize.py yields wrong token for colon
Terry J. Reedy tjre...@udel.edu added the comment: I have not looked at this, but a new parameter would be a new feature. Its a moot point until there is an agreed on patch for a current version. -- nosy: +tjreedy type: behavior - feature request versions: +Python 3.2 -Python 2.4, Python 2.5, Python 2.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2134 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2134] function generate_tokens at tokenize.py yields wrong token for colon
A.M. Kuchling li...@amk.ca added the comment: Unfortunately I think this will break many users of tokenize.py. e.g. http://browsershots.googlecode.com/svn/trunk/devtools/pep8/pep8.py has code like: if (token_type == tokenize.OP and text in '([' and ...): If tokenize now returns LPAR, this code will no longer work correctly. Tools/i18n/pygettext.py, pylint, WebWare, pyfuscate, all have similar code. So I think we can't change the API this radically. Adding a parameter to enable more precise handling of tokens, and defaulting it to off, is probably the only way to change this. -- nosy: +akuchling ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2134 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2134] function generate_tokens at tokenize.py yields wrong token for colon
Guilherme Polo added the comment: I'm attaching a patch that solves this and updates tests. Added file: http://bugs.python.org/file9459/tokenize_r60884.diff __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2134 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2134] function generate_tokens at tokenize.py yields wrong token for colon
New submission from Guilherme Polo: function generate_tokes at tokenize.py yields token OP (51) for colon, while it should be token COLON (11). It probably affects other python versions as well. I'm attaching a minor sample that demonstrates this, running it returns the following output: 1 'if' (1, 0) (1, 2) if a == 2: 1 'a' (1, 3) (1, 4) if a == 2: 51 '==' (1, 5) (1, 7) if a == 2: 2 '2' (1, 8) (1, 9) if a == 2: 51 ':' (1, 9) (1, 10) if a == 2: 1 'print' (2, 0) (2, 5) print 'hey' 3 'hey' (2, 6) (2, 11) print 'hey' 0 '' (3, 0) (3, 0) I didn't check if there are problems with other tokens, I noticed this with colon because I was trying to make some improvements on tabnanny. -- components: Library (Lib) files: tokenize_sample.py messages: 62509 nosy: gpolo severity: normal status: open title: function generate_tokens at tokenize.py yields wrong token for colon type: behavior versions: Python 2.4, Python 2.5, Python 2.6 Added file: http://bugs.python.org/file9452/tokenize_sample.py __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2134 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
