Adam Bartoš added the comment: It could still point to the first or the last byte of the invalid token rather than to the start of the next token. Also, by the Python implementation of the tokenizer in tokenize module we get an ERRORTOKEN containing a non-breaking space followed by a number token containing 2.
---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26152> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
