[issue26843] tokenize does not include Other_ID_Start or Other_ID_Continue in identifier

Joshua Landau added the comment: Sorry, I'd stumbled on my old comment on the closed issue and completely forgot about the *last* time I did the same thing. -- ___ Python tracker

[issue26843] tokenize does not include Other_ID_Start or Other_ID_Continue in identifier

Serhiy Storchaka added the comment: This is a duplicate of issue24194. Yes, there is no progress still. -- nosy: +serhiy.storchaka resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or

[issue26843] tokenize does not include Other_ID_Start or Other_ID_Continue in identifier

New submission from Joshua Landau: This is effectively a continuation of https://bugs.python.org/issue9712. The line in Lib/tokenize.py Name = r'\w+' must be changed to a regular expression that accepts Other_ID_Start at the start and Other_ID_Continue elsewhere. Hence tokenize does not