New submission from Tyler Crompton:
Line 402 in lib/python3.3/tokenize.py, contains the following line:
if first.startswith(BOM_UTF8):
BOM_UTF8 is a bytes object. str.startswith does not accept bytes objects. I was
able to use tokenize.tokenize only after making the following changes:
R. David Murray added the comment:
The docs could certainly be more explicit...currently they state that tokenize
is *detecting* the encoding of the file, which *implies* but does not make
explicit that the input must be binary, not text.
The doc problem will get fixed as part of the fix to
