Berker Peksag added the comment: Thank you for the report, but this behavior is already documented at https://docs.python.org/3/library/tokenize.html
To simplify token stream handling, all Operators and Delimiters tokens are returned using the generic token.OP token type. The exact type can be determined by checking the exact_type property on the named tuple returned from tokenize.tokenize(). If you replace the following line print(ellipsis) with print(ellipsis.exact_type) you will see that it prints '52'. ---------- nosy: +berker.peksag resolution: -> not a bug stage: -> resolved status: open -> closed type: -> behavior _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31394> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com