New submission from Joshua Landau: This is valid:
℘· = 1 print(℘·) #>>> 1 But this gives an error token: from io import BytesIO from tokenize import tokenize stream = BytesIO("℘·".encode("utf-8")) print(*tokenize(stream.read), sep="\n") #>>> TokenInfo(type=56 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') #>>> TokenInfo(type=53 (ERRORTOKEN), string='℘', start=(1, 0), end=(1, 1), line='℘·') #>>> TokenInfo(type=53 (ERRORTOKEN), string='·', start=(1, 1), end=(1, 2), line='℘·') #>>> TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') This is a continuation of http://bugs.python.org/issue9712. I'm not able to reopen the issue, so I thought I should report it anew. It is tokenize that is wrong - Other_ID_Start and Other_ID_Continue are documented to be valid: https://docs.python.org/3.5/reference/lexical_analysis.html#identifiers ---------- components: Library (Lib) messages: 243188 nosy: Joshua.Landau priority: normal severity: normal status: open title: tokenize yield an ERRORTOKEN if an identifier uses Other_ID_Start or Other_ID_Continue type: behavior versions: Python 3.4 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24194> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com