[issue1634034] Show "expected" token on syntax error

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Closing as per above

--
resolution:  -> out of date
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I think that this issue should be closed as 'out of date' as it was pretty 
open-ended and it is unclear what request remains.

For the specific case "for a in (8,9)", the suggested "expected ':'" has been 
added on another issue.  I expect that there are other additions from other 
issues.

For "for a in (8,9: print a," there is no change but for "for a in (8,9]" we 
now have "closing parenthesis ']' does not match opening parenthesis '('".  
This properly does not say that "expected ')'" as the needed fix might be to 
insert opener '['.

For many other cases, the proposed additions were disputed as not helpful, 
mostly because multiple things could be expected.  I think other suggestions, 
based on current master, should be new issues.

--
nosy: +pablogsal, terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +6147
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show "expected" token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Similar enhancement has been implemented in PyPy just now.

https://morepypy.blogspot.de/2018/04/improving-syntaxerror-in-pypy.html

--
versions: +Python 3.8 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I agree, the main problem is in the fact that expected token is not always 
singular. And even most expected token is a little subjective. The better 
solution will be to expect several possible tokens. This requires some parser 
modification.

--
stage: patch review - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Hmm, expected attribute is set when there is only one possible expected token 
in PyParser_AddToken(). I don't understand why error messages are so misleading 
for def f(*23): (here not only ')', but a name possible).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

   dict(a = i for i in range(10))
 +SyntaxError: invalid syntax - ')' expected

 The () are ok, the message is misleading.

dict(a = i) is valid syntax, the compiler expects ) instead of invalid 
for.

 'name' here is a bit vague.

The compiler actually expects a name (using Python terminology, see for example 
NameError). Of course you can propose an other name for name (this is just an 
entity in _PyParser_TokenDescs array).

   def f(x, None):
  ... pass
 +SyntaxError: invalid syntax - ')' expected

   def f(*None):
  ... pass
 +SyntaxError: invalid syntax - ')' expected

 Here the () are ok too.

The compiler means def f(x,) and def f(*), not def f() as you possible 
expects.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Ezio Melotti]

Ezio Melotti added the comment:

I'm not saying that these errors are wrong -- just that they are misleading 
(i.e. they might lead the user on the wrong path, and make finding the actual 
problem more difficult).

It should be noted that the examples I pasted don't include a full traceback 
though.  The presence of the caret (^) in the right place will definitely make 
things clearer.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Serhiy Storchaka]

Changes by Serhiy Storchaka storch...@gmail.com:


--
stage: test needed - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Patch updated (thanks Benjamin for comments).

--
Added file: http://bugs.python.org/file27708/syntax-error-hints-3.4_2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Ezio Melotti]

Ezio Melotti added the comment:

Looking at the changes in the patch it seems to me that, in at least a few 
cases, it's better to have a bare invalid syntax than a misleading error.
For example:

  dict(a = i for i in range(10))
+SyntaxError: invalid syntax - ')' expected

The () are ok, the message is misleading.


  obj.None = 1
+SyntaxError: invalid syntax - name expected

'name' here is a bit vague.

 
  def f(x, None):
 ... pass
+SyntaxError: invalid syntax - ')' expected

  def f(*None):
 ... pass
+SyntaxError: invalid syntax - ')' expected

Here the () are ok too.

 
  def f(**None):
 ... pass
+SyntaxError: invalid syntax - name expected

Here I would have expected the ')' expected like in the previous example, but 
there's name instead (which is a bit better, albeit inconsistent).


I wouldn't consider this an improvement, but for other situations the error 
message is probably useful.
I see 3 options here:
1) we find a way to show the expected token only when the message is not 
misleading;
2) if the expected token is useful more often than it is misleading, then we 
could apply the patch as is;
3) if it is misleading more often than it is useful, it's probably better to 
reject the patch.

--
nosy: +ezio.melotti

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I'm attaching a new version of the patch, based on Dave's (from 2.5 years ago). 
This patch is against the 3.4.

Previous patches contained an error in the message formatting. buf variable 
out of scope before msg used. Appending '\0' to the format string isn't 
guaranteed to NUL-terminate the string. Actually it do nothing (except 
producing a warning).

--
nosy: +serhiy.storchaka
versions: +Python 3.4 -Python 3.2
Added file: http://bugs.python.org/file27530/syntax-error-hints-3.4.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

+1 on the basic idea to make error messages more informative where possible, 
but am not sure how it would work in any but the more simple cases.  

How would work in cases where there are multiple possible expected tokens?

def f(x 3):  
   SyntaxError: invalid syntax

It chokes at the 3.  The expected token is either a comma, colon, or closing 
parenthesis.

Also, the most annoying and least obvious syntax errors are ones that are 
revealed many characters away from the original cause (i.e. unbalanced opening 
brackets or parentheses).  Am not sure how you can readily construct a helpful 
message in these cases.

--
nosy: +rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Terry J. Reedy]

Changes by Terry J. Reedy tjre...@udel.edu:


--
versions:  -Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +benjamin.peterson
versions: +Python 3.2 -Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1634034] Show expected token on syntax error

[Dave Malcolm]

Dave Malcolm dmalc...@redhat.com added the comment:

I'm attaching a new version of the patch, based on Oliver's (from 3 years ago). 
 This patch is against the py3k branch.

I've introduced a new table of (const) strings: _PyParser_TokenDescs, giving 
descriptions of each token type, so that you get e.g. ')' rather than RPAR

The patch of pythonrun.c is unchanged other than using the description table, 
rather than the name table.

I've patched the expected results for the doctests in test_genexps and 
test_syntax.py so that these files pass: this gives the code the beginnings of 
a test suite.

The existing patch adds this compiler warning for me (gcc 4.4.2, on Fedora 12):
  Python/pythonrun.c: In function ‘err_input’:
  Python/pythonrun.c:1923: warning: embedded ‘\0’ in format
However I believe that snprintf isn't guaranteed to NUL-terminate the string 
under all conditions on all platforms, so the '\0' seems a sane failsafe.

How does this look?

I haven't attempted to deal with places where multiple token types are 
permitted, and it does sometimes simply emit invalid syntax.

--
keywords: +patch
nosy: +dmalcolm
Added file: http://bugs.python.org/file15860/syntax-error-hints.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue1634034
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com