[issue34515] lib2to3: support non-ASCII identifiers

miss-islington added the comment: New changeset 51dbae867e82014f9af89662977e4981463c51e8 by Miss Islington (bot) in branch '3.7': closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950) https://github.com/python/cpython/commit/51dbae867e82014f9af89662977e4981463c51e8

[issue34515] lib2to3: support non-ASCII identifiers

Benjamin Peterson added the comment: New changeset 10a428b64b3f224e2ccd40ff2afb141b9b3425b1 by Benjamin Peterson (Monson Shao) in branch 'master': closes bpo-34515: Support non-ASCII identifiers in lib2to3. (GH-8950)

[issue34515] lib2to3: support non-ASCII identifiers

Change by miss-islington : -- pull_requests: +8756 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34515] lib2to3: support non-ASCII identifiers

Change by monson : -- versions: +Python 3.8 -Python 3.7 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue34515] lib2to3: support non-ASCII identifiers

Change by monson : -- keywords: +patch pull_requests: +8426 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list

[issue34515] lib2to3: support non-ASCII identifiers

New submission from monson : Python 3.0 introduces additional characters from outside the ASCII range (see PEP 3131). see https://docs.python.org/3/reference/lexical_analysis.html#identifiers But lib2to3 can't tokenize them corretly. ``` $ echo '中 = 1' | python3.7 -m lib2to3.pgen2.tokenize