[issue1634034] Show "expected" token on syntax error

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Closing as per above

--
resolution:  -> out of date
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I think that this issue should be closed as 'out of date' as it was pretty 
open-ended and it is unclear what request remains.

For the specific case "for a in (8,9)", the suggested "expected ':'" has been 
added on another issue.  I expect that there are other additions from other 
issues.

For "for a in (8,9: print a," there is no change but for "for a in (8,9]" we 
now have "closing parenthesis ']' does not match opening parenthesis '('".  
This properly does not say that "expected ')'" as the needed fix might be to 
insert opener '['.

For many other cases, the proposed additions were disputed as not helpful, 
mostly because multiple things could be expected.  I think other suggestions, 
based on current master, should be new issues.

--
nosy: +pablogsal, terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +6147
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Similar enhancement has been implemented in PyPy just now.

https://morepypy.blogspot.de/2018/04/improving-syntaxerror-in-pypy.html

--
versions: +Python 3.8 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Hmm, "expected" attribute is set when there is only one possible expected token 
in PyParser_AddToken(). I don't understand why error messages are so misleading 
for "def f(*23):" (here not only ')', but a name possible).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I agree, the main problem is in the fact that "expected token" is not always 
singular. And even "most expected token" is a little subjective. The better 
solution will be to expect several possible tokens. This requires some parser 
modification.

--
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Ezio Melotti]

Ezio Melotti added the comment:

I'm not saying that these errors are wrong -- just that they are misleading 
(i.e. they might lead the user on the wrong path, and make finding the actual 
problem more difficult).

It should be noted that the examples I pasted don't include a full traceback 
though.  The presence of the caret (^) in the right place will definitely make 
things clearer.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

>  >>> dict(a = i for i in range(10))
> +SyntaxError: invalid syntax - ')' expected
>
> The () are ok, the message is misleading.

"dict(a = i)" is valid syntax, the compiler expects ")" instead of invalid 
"for".

> 'name' here is a bit vague.

The compiler actually expects a name (using Python terminology, see for example 
NameError). Of course you can propose an other name for "name" (this is just an 
entity in _PyParser_TokenDescs array).

>  >>> def f(x, None):
>  ... pass
> +SyntaxError: invalid syntax - ')' expected
>
>  >>> def f(*None):
>  ... pass
> +SyntaxError: invalid syntax - ')' expected
>
> Here the () are ok too.

The compiler means "def f(x,)" and "def f(*)", not "def f()" as you possible 
expects.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Ezio Melotti]

Ezio Melotti added the comment:

Looking at the changes in the patch it seems to me that, in at least a few 
cases, it's better to have a bare "invalid syntax" than a misleading error.
For example:

 >>> dict(a = i for i in range(10))
+SyntaxError: invalid syntax - ')' expected

The () are ok, the message is misleading.


 >>> obj.None = 1
+SyntaxError: invalid syntax - name expected

'name' here is a bit vague.

 
 >>> def f(x, None):
 ... pass
+SyntaxError: invalid syntax - ')' expected

 >>> def f(*None):
 ... pass
+SyntaxError: invalid syntax - ')' expected

Here the () are ok too.

 
 >>> def f(**None):
 ... pass
+SyntaxError: invalid syntax - name expected

Here I would have expected the "')' expected" like in the previous example, but 
there's "name" instead (which is a bit better, albeit inconsistent).


I wouldn't consider this an improvement, but for other situations the error 
message is probably useful.
I see 3 options here:
1) we find a way to show the expected token only when the message is not 
misleading;
2) if the expected token is useful more often than it is misleading, then we 
could apply the patch as is;
3) if it is misleading more often than it is useful, it's probably better to 
reject the patch.

--
nosy: +ezio.melotti

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Patch updated (thanks Benjamin for comments).

--
Added file: http://bugs.python.org/file27708/syntax-error-hints-3.4_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
stage: test needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I'm attaching a new version of the patch, based on Dave's (from 2.5 years ago). 
This patch is against the 3.4.

Previous patches contained an error in the message formatting. "buf" variable 
out of scope before "msg" used. Appending '\0' to the format string isn't 
guaranteed to NUL-terminate the string. Actually it do nothing (except 
producing a warning).

--
nosy: +serhiy.storchaka
versions: +Python 3.4 -Python 3.2
Added file: http://bugs.python.org/file27530/syntax-error-hints-3.4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Raymond Hettinger]

Raymond Hettinger  added the comment:

+1 on the basic idea to make error messages more informative where possible, 
but am not sure how it would work in any but the more simple cases.  

How would work in cases where there are multiple possible "expected" tokens?

   >>> def f(x 3):  
   SyntaxError: invalid syntax

It chokes at the "3".  The expected token is either a comma, colon, or closing 
parenthesis.

Also, the most annoying and least obvious syntax errors are ones that are 
revealed many characters away from the original cause (i.e. unbalanced opening 
brackets or parentheses).  Am not sure how you can readily construct a helpful 
message in these cases.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Terry J. Reedy]

Changes by Terry J. Reedy :


--
versions:  -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +benjamin.peterson
versions: +Python 3.2 -Python 3.1

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Dave Malcolm]

Dave Malcolm  added the comment:

I'm attaching a new version of the patch, based on Oliver's (from 3 years ago). 
 This patch is against the py3k branch.

I've introduced a new table of (const) strings: _PyParser_TokenDescs, giving 
descriptions of each token type, so that you get e.g. "')'" rather than "RPAR"

The patch of pythonrun.c is unchanged other than using the description table, 
rather than the name table.

I've patched the expected results for the doctests in test_genexps and 
test_syntax.py so that these files pass: this gives the code the beginnings of 
a test suite.

The existing patch adds this compiler warning for me (gcc 4.4.2, on Fedora 12):
  Python/pythonrun.c: In function ‘err_input’:
  Python/pythonrun.c:1923: warning: embedded ‘\0’ in format
However I believe that snprintf isn't guaranteed to NUL-terminate the string 
under all conditions on all platforms, so the '\0' seems a sane failsafe.

How does this look?

I haven't attempted to deal with places where multiple token types are 
permitted, and it does sometimes simply emit "invalid syntax".

--
keywords: +patch
nosy: +dmalcolm
Added file: http://bugs.python.org/file15860/syntax-error-hints.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Daniel Diniz]

Daniel Diniz  added the comment:

Sounds really useful.

--
nosy: +ajaksu2
stage:  -> test needed
versions: +Python 2.7, Python 3.1 -Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com