[issue26843] tokenize does not include Other_ID_Start or Other_ID_Continue in identifier

[Joshua Landau]

Joshua Landau added the comment:

Sorry, I'd stumbled on my old comment on the closed issue and completely forgot 
about the *last* time I did the same thing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26843] tokenize does not include Other_ID_Start or Other_ID_Continue in identifier

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This is a duplicate of issue24194. Yes, there is no progress still.

--
nosy: +serhiy.storchaka
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> tokenize yield an ERRORTOKEN if an identifier uses 
Other_ID_Start or Other_ID_Continue

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26843] tokenize does not include Other_ID_Start or Other_ID_Continue in identifier

[Joshua Landau]

New submission from Joshua Landau:

This is effectively a continuation of https://bugs.python.org/issue9712.

The line in Lib/tokenize.py

Name = r'\w+'

must be changed to a regular expression that accepts Other_ID_Start at the 
start and Other_ID_Continue elsewhere. Hence tokenize does not accept '℘·'.


See the reference here:

https://docs.python.org/3.5/reference/lexical_analysis.html#identifiers

I'm unsure whether unicode normalization (aka the `xid` properties) needs to be 
dealt with too.


Credit to toriningen from http://stackoverflow.com/a/29586366/1763356.

--
components: Library (Lib)
messages: 264145
nosy: Joshua.Landau
priority: normal
severity: normal
status: open
title: tokenize does not include Other_ID_Start or Other_ID_Continue in 
identifier
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com