[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Irit Katriel]

Change by Irit Katriel :


--
resolution:  -> wont fix
stage: needs patch -> resolved
status: open -> closed
superseder:  -> Close 2to3 issues and list them here

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Justin Arthur]

Justin Arthur  added the comment:

Not sure if there is consensus on how to fix, but fixing #12731 will fix this 
for most of the cases I've seen complaints about as a side effect.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Batuhan]

Batuhan  added the comment:

Is there a consensus about fixing this? By the way, this isn't valid in the 
current tokenizer too. 
1,0-1,2:NAME   'iÌ'
1,2-1,3:ERRORTOKEN '‡'
1,4-1,5:OP '='
1,6-1,7:NUMBER '5'
1,7-1,8:NEWLINE'\n'

--
nosy: +BTaskaya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Ned Deily]

Change by Ned Deily :


--
nosy:  -ned.deily
stage: resolved -> needs patch
versions: +Python 3.9 -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Benjamin Peterson]

Change by Benjamin Peterson :


--
resolution: not a bug -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Ned Deily]

Ned Deily  added the comment:

> 2to3 should be able to parse valid Python 3 code.

OK, then should the original behavior here be treated as a bug and fixed?  If 
so, this issue should be re-opened.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Benjamin Peterson]

Benjamin Peterson  added the comment:

2to3 should be able to parse valid Python 3 code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Ned Deily]

Ned Deily  added the comment:

Benjamin, can you answer Justin's question above?

--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Justin Arthur]

Justin Arthur  added the comment:

Ned, can you confirm that 2to3 is not intended for cumulative/incremental runs 
over the same codebase?

If it's not intended to be run on previously ported code, this will just need 
to be fixed on the lib2to3 downstream projects like awpa and Black that are 
encountering this issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Ned Deily]

Ned Deily  added the comment:

"2to3 is a Python program that reads Python 2.x source code and applies a 
series of fixers to transform it into valid Python 3.x code."

The example you supply, badvar,py, is not a valid Python 2.x program.  Python 2 
identifiers cannot contain such characters.

https://docs.python.org/3/library/2to3.html
https://docs.python.org/2/reference/lexical_analysis.html#identifiers
https://docs.python.org/3/reference/lexical_analysis.html#identifiers

--
nosy: +ned.deily
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38032] lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks

[Justin ARthur]

New submission from Justin ARthur :

Python 3 code with an identifier that has a non-spacing mark in it does not get 
tokenized by lib2to3 and will result in an exception thrown in the parsing 
process.

Parsing the attached file (badvar.py), results in `ParseError: bad token: 
type=58, value='̇', context=('', (1, 1))`

This happens because the Name pattern regular expression in lib2to3 is `r'\w+'` 
and the word character class doesn't contain non-spacing marks (and possible 
other [continuation characters allowed in Python 3 
identifiers](https://docs.python.org/3/reference/lexical_analysis.html#identifiers)).

(reported by energizer in the Python IRC channel)

--
components: 2to3 (2.x to 3.x conversion tool), Library (Lib)
files: badvar.py
messages: 351153
nosy: JustinTArthur
priority: normal
severity: normal
status: open
title: lib2to3 doesn't parse Python 3 identifiers containing non-spacing marks
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file48592/badvar.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com