[issue20608] 'SyntaxError: invalid token' is unfriendly

Terry J. Reedy added the comment: Fixed elsewhere. >>> 04208 File "", line 1 04208 ^ SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers >>> 0o38 File "", line 1 0o38 ^ SyntaxError: invalid digit '8' in

[issue20608] 'SyntaxError: invalid token' is unfriendly

John Parejko added the comment: I had filed issue 29146 but eventually found this, which has patches to fix the exception messages. Could someone please look at this and get it incorporated into python? -- ___ Python tracker

[issue20608] 'SyntaxError: invalid token' is unfriendly

Changes by John Parejko : -- nosy: +John Parejko ___ Python tracker ___ ___

[issue20608] 'SyntaxError: invalid token' is unfriendly

Changes by Berker Peksag : -- versions: +Python 3.7 -Python 3.5 ___ Python tracker ___

[issue20608] 'SyntaxError: invalid token' is unfriendly

Marius Gedminas added the comment: Oh, hey, PEP 3127 actually asks for a better error message than invalid token for this case: http://www.python.org/dev/peps/pep-3127/#tokenizer-exception-handling So here's a tentative patch to test the waters. I still haven't figured out how to write

[issue20608] 'SyntaxError: invalid token' is unfriendly

Marius Gedminas added the comment: I resolved my compilation problems (by running 'make distclean'). There are some problems with my patch: - leading is misspelled (as lleading) - literals like 0x1z, 0o18, 0b12, 1.2e-1x produce a generic invalid syntax message instead of the specific bad digit

[issue20608] 'SyntaxError: invalid token' is unfriendly

Marius Gedminas added the comment: I see that I misunderstood Serhiy's comment. I assumed he meant the caret will be pointing to the 1st digit that is invalid. Instead what actually happens is that E_TOKEN is emitted only if the 1st digit after the 0x/0o/0b prefix is invalid. So, I get the

[issue20608] 'SyntaxError: invalid token' is unfriendly

Marius Gedminas added the comment: Here's version 2 of the patch: - spelling error fixed - 0b2, 0o8, 0xg, 0e-x show the expected error at the expected place - 0b02, 0o08, 0x0g, 0e-0x continue produce a generic syntax error because the tokenizer thinks these are a pair of valid tokens (0b0

[issue20608] 'SyntaxError: invalid token' is unfriendly

Marius Gedminas added the comment: Version 3 of the patch catches bad digits in the middle of a literal, like this: 0o01010118001 File stdin, line 1 0o01010118001 ^ SyntaxError: bad digit in octal literal -- Added file:

[issue20608] 'SyntaxError: invalid token' is unfriendly

Marius Gedminas added the comment: Here are some unit tests for the new syntax errors (in test_syntax.py; test_tokenize.py turned out to be totally unrelated). One possible shortcoming: they do not test the column of the syntax error. -- Added file:

[issue20608] 'SyntaxError: invalid token' is unfriendly

Marius Gedminas added the comment: Updated test that checks the syntax error offset as well. I think I'm done with the iterations. I'll be waiting for feedback. -- Added file: http://bugs.python.org/file34094/better-errors-test-v2.patch ___ Python

[issue20608] 'SyntaxError: invalid token' is unfriendly

Serhiy Storchaka added the comment: Marius, you can remove unneeded patches (click on the edit link and then press the Unlink button). -- stage: - patch review type: - enhancement ___ Python tracker rep...@bugs.python.org

[issue20608] 'SyntaxError: invalid token' is unfriendly

Benjamin Peterson added the comment: I can look at this when the time for 3.5 rolls around. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20608 ___

[issue20608] 'SyntaxError: invalid token' is unfriendly

New submission from Marius Gedminas: Type something like the following at the interpreter prompt: 04208 File stdin, line 1 04208 ^ SyntaxError: invalid token This is not very descriptive. I suggest SyntaxError: invalid octal digit. -- components: Interpreter Core

[issue20608] 'SyntaxError: invalid token' is unfriendly

Marius Gedminas added the comment: I was looking at the current hg tip. The lexer emits E_TOKEN errors for the following cases: - invalid hex digit - invalid octal digit - invalid binary digit - invalid digit in float exponent - old-style octal constant (e.g. 001), which is no longer accepted

[issue20608] 'SyntaxError: invalid token' is unfriendly

Serhiy Storchaka added the comment: Note that invalid token is emitted only on invalid first digit: 0b2 File stdin, line 1 0b2 ^ SyntaxError: invalid token 0b02 File stdin, line 1 0b02 ^ SyntaxError: invalid syntax See also issue1634034. -- nosy: