New submission from Brian Bossé <pen...@gmail.com>: Executing the following code against a py file which contains line continuations generates an assert: import tokenize foofile = open(filename, "r") tokenize.untokenize(list(tokenize.generate_tokens(foofile.readline)))
(note, the list() is important due to issue #8478) The assert triggered is: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Python27\lib\tokenize.py", line 262, in untokenize return ut.untokenize(iterable) File "C:\Python27\lib\tokenize.py", line 198, in untokenize self.add_whitespace(start) File "C:\Python27\lib\tokenize.py", line 187, in add_whitespace assert row <= self.prev_row AssertionError I have tested this in 2.6.5, 2.7 and 3.1.2. The line numbers may differ but the stack is otherwise identical between these versions. Example input code: foo = \ 3 If the assert is removed, the code generated is still incorrect. For example, the input: foo = 3 if foo == 5 or \ foo == 1 pass becomes: foo = 3 if foo == 5 orfoo == 1 pass which besides not having the line continuation, is functionally incorrect. I'm wrapping my head around the functionality of this module and am willing to do the legwork to get a fix in. Ideas on how to go about it are more than welcome. Ironic aside: this bug is present when tokenize.py itself is used as input. ---------- components: Library (Lib) messages: 117538 nosy: Brian.Bossé priority: normal severity: normal status: open title: tokenizer.untokenize not invariant with line continuations type: behavior versions: Python 2.6, Python 2.7, Python 3.1 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue9974> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com