[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Change by Irit Katriel : -- resolution: -> wont fix stage: needs patch -> resolved status: open -> closed superseder: -> Close 2to3 issues and list them here ___ Python tracker _

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Justin Arthur added the comment: Not sure if there is consensus on how to fix, but fixing #12731 will fix this for most of the cases I've seen complaints about as a side effect. -- ___ Python tracker __

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Batuhan added the comment: Is there a consensus about fixing this? By the way, this isn't valid in the current tokenizer too. 1,0-1,2:NAME 'iÌ' 1,2-1,3:ERRORTOKEN '‡' 1,4-1,5:OP '=' 1,6-1,7:NUMBER '5' 1,7-1,8:

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Change by Ned Deily : -- nosy: -ned.deily stage: resolved -> needs patch versions: +Python 3.9 -Python 3.5, Python 3.6 ___ Python tracker ___ _

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Change by Benjamin Peterson : -- resolution: not a bug -> status: closed -> open ___ Python tracker ___ ___ Python-bugs-list mailin

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Ned Deily added the comment: > 2to3 should be able to parse valid Python 3 code. OK, then should the original behavior here be treated as a bug and fixed? If so, this issue should be re-opened. -- ___ Python tracker

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Benjamin Peterson added the comment: 2to3 should be able to parse valid Python 3 code. -- ___ Python tracker ___ ___ Python-bugs-li

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Ned Deily added the comment: Benjamin, can you answer Justin's question above? -- nosy: +benjamin.peterson ___ Python tracker ___ _

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Justin Arthur added the comment: Ned, can you confirm that 2to3 is not intended for cumulative/incremental runs over the same codebase? If it's not intended to be run on previously ported code, this will just need to be fixed on the lib2to3 downstream projects like awpa and Black that are e

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

Ned Deily added the comment: "2to3 is a Python program that reads Python 2.x source code and applies a series of fixers to transform it into valid Python 3.x code." The example you supply, badvar,py, is not a valid Python 2.x program. Python 2 identifiers cannot contain such characters. ht

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

New submission from Justin ARthur : Python 3 code with an identifier that has a non-spacing mark in it does not get tokenized by lib2to3 and will result in an exception thrown in the parsing process. Parsing the attached file (badvar.py), results in `ParseError: bad token: type=58, value='̇',