Brian Bossé pen...@gmail.com added the comment:
Yup, that's related to ENDMARKER being tokenized to its own line, even if EOF
happens at the end of the last line of actual code. I don't know if anything
relies on that behavior so I can't really suggest changing it.
My patch handles
Brian Bossé pen...@gmail.com added the comment:
No idea if I'm getting the patch format right here, but tally ho!
This is keyed from release27-maint
Index: Lib/tokenize.py
===
--- Lib/tokenize.py (revision 85136)
+++ Lib
New submission from Brian Bossé pen...@gmail.com:
Executing the following code against a py file which contains line
continuations generates an assert:
import tokenize
foofile = open(filename, r)
tokenize.untokenize(list(tokenize.generate_tokens(foofile.readline)))
(note, the list
