[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: @Serhiy/anyone: can I get another review, so that we can commit this in time for beta? Thanks! -- ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: Thanks for the detailed review, Serhiy! Next try incoming. -- Added file: http://bugs.python.org/file42939/numeric_underscores_final_v8.diff ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Eric V. Smith added the comment: I've created issue 27080 to track the formatting part of this. -- ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Stefan Krah added the comment: Thanks, Georg! The decimal parts look good to me. I understand that people wonder about the relaxed rules for Decimal -- we have discussed that here: https://mail.python.org/pipermail/python-dev/2016-March/143557.html I don't think that it will be a problem

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Serhiy Storchaka added the comment: Added comments on Rietveld. -- ___ Python tracker ___ ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: Thanks Eric! Serhiy, do you want to do a review? The v6/v7 patches are based on your "strict" patch with the constructor changes adapted from v4. New version v7 addresses the review comments from Stefan and Martin. -- Added file:

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Eric V. Smith added the comment: Yes, I'll read PEP 515 and work on the formatting. -- ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: Note: the changes for format()ting ("_" as thousands separator) are still missing. Eric, would you consider doing this part? -- nosy: +eric.smith ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Changes by Georg Brandl : Removed file: http://bugs.python.org/file41892/numeric_underscores_v2.diff ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Changes by Georg Brandl : Removed file: http://bugs.python.org/file42852/numeric_underscores_final_v5.diff ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Changes by Georg Brandl : Removed file: http://bugs.python.org/file41888/numeric_underscores.diff ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Changes by Georg Brandl : Removed file: http://bugs.python.org/file41894/numeric_underscores_v3_full.diff ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Changes by Georg Brandl : Added file: http://bugs.python.org/file42854/numeric_underscores_final_v6.diff ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: New patch; implements the accepted version of the PEP. I added the additional tests, thanks Stefan! -- Added file: http://bugs.python.org/file42852/numeric_underscores_final_v5.diff ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Stefan Behnel added the comment: Ah, thanks. Here's my implementation then: https://github.com/cython/cython/pull/499/files It seems that tests for valid complex literals are missing. I've added these to the end of the list: '1_00_00.5j', '1_00_00.5e5', '1_00_00j',

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: The last patch isn't up to date with the PEP; Serhiy's patch is the closest one. -- ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Stefan Behnel added the comment: Nice one. While reimplementing it for Cython, I noticed that the grammar described in the PEP isn't exactly as it's implemented, though. The grammar says digit (["_"] digit)* whereas the latest patch (v4) says `digit` (`digit` | "_")* and also

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Stefan Krah added the comment: I like the feature for literals, but I'm not sure about conversions from string. It slows down the conversion for (IMO) a very small benefit. Other languages allow it, but I've never attempted to use the feature: $ ocaml OCaml version 4.02.1 #

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: It's mostly for consistency. For example, ``int(x, 0)`` is defined by the docs as "interpret x as in a literal". Other bases have special cases as well, e.g. "0x" is accepted by base 16. In the current version of the conversions, the string is scanned for "_"

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Stefan Krah added the comment: If the string conversions stay, may I suggest two functions: 1) PyUnicode_NumericAsAscii() 2) PyUnicode_NumericAsAsciiWS() The first one eliminates only underscores, the second one both underscores and leading/trailing whitespace. Decimal must support both:

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Stefan Krah added the comment: Correction: The explanation of the functions should be reversed. -- ___ Python tracker ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: Thanks, I hadn't looked at cdecimal yet - I was planning to ask you to do the necessary changes there :) But there are a few versions of this (e.g. converting unicode digits to ASCII) scattered throughout the codebase, it would make sense to consolidate on this

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Stefan Krah added the comment: > Georg Brandl added the comment: > > Thanks, I hadn't looked at cdecimal yet - I was planning to ask you to do the > necessary changes there :) Oh, well. :) > But there are a few versions of this (e.g. converting unicode digits to > ASCII) scattered

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Stefan Krah added the comment: I still wonder about the complexity of all this for decimal. We now have two grammars on top of each other, this being the actual one for decimal: http://speleotrove.com/decimal/daconvs.html For string conversions I'd prefer a lax way (similar to OCaml) that

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: Hm. On the one hand there is a spec, so it can be argued that underscores don't belong to Decimal. On the other hand, if we get Decimal literals at one point, there will be a strong argument for allowing underscores in them as in all other number literals.

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: Raymond, you've also worked on Decimal - do you have an opinion on allowing underscores in Decimal(string) conversions? -- nosy: +rhettinger ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: New patch matching revision of PEP. -- Added file: http://bugs.python.org/file41892/numeric_underscores_v2.diff ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Serhiy Storchaka added the comment: Proposed patch implements strict underscore rules. The implementation is not more complex. -- Added file: http://bugs.python.org/file41893/numeric_underscores_strict.patch ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: New patch with minimal doc updates. -- Added file: http://bugs.python.org/file41896/numeric_underscores_v4_full.diff ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Changes by Mark Dickinson : -- nosy: +mark.dickinson ___ Python tracker ___ ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: This patch includes int(), float(), complex() operations, as well as _pydecimal. -- Added file: http://bugs.python.org/file41894/numeric_underscores_v3_full.diff ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Serhiy Storchaka added the comment: * Java: only between digits. [1] * Julia: only between digits. [2] (not well specified) * C# 7.0 (proposal): only between digits, but adjacent underscores allowed. [3] * Ada: only between digits. [4] (strong but very simple rules) * D: very much like proposed

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: It sure is more strict, but I don't think it's simpler (and it's definitely not simpler to implement). (Also 1_j is pretty nice, I wouldn't want to lose that.) We can also check what other languages do. * Rust: very much like this, but trailing underscores

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Changes by Ethan Furman : -- nosy: +ethan.furman ___ Python tracker ___ ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Serhiy Storchaka added the comment: C++14 uses the same strict rule as Ada, but uses apostrphes instead of underscores. [1] Thus there are two groups of languages, implementing strict or lenient rules: * Strict: Ada, C++, Java, C#, Ruby, Julia, Perl (as documented), Swift (textual

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Yury Selivanov added the comment: > I prefer simpler and more strict rule: > * Underscores are allowed only between digits in numeric literals. +1. But in any case we need a PEP for this change. -- nosy: +yselivanov ___ Python tracker

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Georg Brandl added the comment: PEP 515 is written up and posted to python-dev. -- ___ Python tracker ___ ___

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

New submission from Georg Brandl: As discussed on python-ideas: https://mail.python.org/pipermail/python-ideas/2016-February/038354.html The rules are: Underscores are allowed anywhere in numeric literals, except: * at the beginning of a literal (obviously) * at the end of a literal *

[issue26331] Tokenizer: allow underscores for grouping in numeric literals

Petr Viktorin added the comment: Regarding the patch: if trailing underscores are not allowed, `0 if 1_else 1` should be illegal. -- nosy: +encukou ___ Python tracker