[issue19035] tokenize.generate_tokens treat '\f' symbol as the end of file (when reading in unicode)

Barry A. Warsaw added the comment: Ha! Apparently this bug broke coverage for the Mailman 3 source code: https://bitbucket.org/ned/coveragepy/issue/360/html-reports-get-confused-by-l-in-the-code -- ___ Python tracker

[issue19035] tokenize.generate_tokens treat '\f' symbol as the end of file (when reading in unicode)

Changes by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue19035] tokenize.generate_tokens treat '\f' symbol as the end of file (when reading in unicode)

R. David Murray added the comment: I suspect this isn't the only place where the change in what is considered a (unicode) line ending character between 2.6 and 2.7/python3 is an issue. As you observe, it causes very subtle bugs. I'm going to have to go trolling through the python3 email pack

[issue19035] tokenize.generate_tokens treat '\f' symbol as the end of file (when reading in unicode)

New submission from Alexey Umnov: I execute the following code on the attached file 'text.txt': import tokenize import codecs with open('text.txt', 'r') as f: reader = codecs.getreader('utf-8')(f) tokens = tokenize.generate_tokens(reader.readline) The file 'text.txt' has the followin