New submission from Guilherme Polo: function generate_tokes at tokenize.py yields token OP (51) for colon, while it should be token COLON (11). It probably affects other python versions as well.
I'm attaching a minor sample that demonstrates this, running it returns the following output: 1 'if' (1, 0) (1, 2) if a == 2: 1 'a' (1, 3) (1, 4) if a == 2: 51 '==' (1, 5) (1, 7) if a == 2: 2 '2' (1, 8) (1, 9) if a == 2: 51 ':' (1, 9) (1, 10) if a == 2: 1 'print' (2, 0) (2, 5) print 'hey' 3 "'hey'" (2, 6) (2, 11) print 'hey' 0 '' (3, 0) (3, 0) I didn't check if there are problems with other tokens, I noticed this with colon because I was trying to make some improvements on tabnanny. ---------- components: Library (Lib) files: tokenize_sample.py messages: 62509 nosy: gpolo severity: normal status: open title: function generate_tokens at tokenize.py yields wrong token for colon type: behavior versions: Python 2.4, Python 2.5, Python 2.6 Added file: http://bugs.python.org/file9452/tokenize_sample.py __________________________________ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2134> __________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
