[issue29051] Improve error reporting involving f-strings (PEP 498)

[Eric V. Smith]

Eric V. Smith  added the comment:

Because this issue describes two different problems, I'm going to close it.

The part of it that involves errors during the evaluation of a syntactically 
valid expression was at least partially fixed in #30465. I will probably 
re-work how this fix was implemented.

I'm working on the syntax error part of it in #34364.

--
dependencies:  -Tracebacks should contain the first line of continuation lines
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Eric V. Smith]

Eric V. Smith  added the comment:

The example above (msg289501) has been fixed. I think it was in #30465.

It also fixes some of these other cases, but I haven't reviewed them all.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Liran Nuna]

Changes by Liran Nuna :


--
nosy: +Liran Nuna

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Mateusz Bysiek]

Changes by Mateusz Bysiek :


--
nosy: +mbdevpl

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Jim Fasarakis-Hilliard]

Changes by Jim Fasarakis-Hilliard :


--
nosy: +Jim Fasarakis-Hilliard

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Claudiu Popa]

Claudiu Popa added the comment:

I'm adding another example here, where the lineno reporting is wrong:

  from ast import parse 
  n = parse('''

  def test():
 return f"{a}"
  ''')
  f = n.body[0].body[0].value.values[0]
  n = f.value
  print("name lineno", n.lineno)

In this example, the line number of the f-string inner variable is 1, while it 
should be 3.
As Mark Shannon said, this bug is affecting tools such as pyflakes and pylint.

--
nosy: +Claudiu.Popa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Mark Shannon]

Mark Shannon added the comment:

It is also worth mentioning that incorrect line numbers means that tools like 
pyflakes, pylint, mypy, lgtm, etc, need to reimplement parsing of f-strings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Mark Shannon]

Mark Shannon added the comment:

This problem is the parsing of f-strings. 

The expressions in an f-string are not "eval"ed in the sense of the eval() 
function. They are evaluated exactly the same as any other Python expression. 
However the parsing of f-strings does not provide correct line numbers.

This problem also manifests itself in the ast and tokenize modules.

>>> m = ast.parse("""f'''
... {
... FOO
... }
... '''
... """)

>>> m.body[0].value.values[1].value.id
'FOO'
>>> m.body[0].value.values[1].value.lineno
2

That 2 should be a 3, and yet
eval(compile(m, "test2", "exec"))
  File "", line 1, in 
  File "test2", line 5, in 
NameError: name 'FOO' is not defined

gives line 5 for the error, so not only are the line numbers wrong they are 
inconsistent.

The problem is that the internals of the f-string are not tokenized and parsed 
using the normal mechanism, but in an ad-hoc fashion in Python-ast.c as 
demonstrated when tokenizing the source

$ python3.6 -m tokenize test2
0,0-0,0:ENCODING   'utf-8'
1,0-5,3:STRING "f'''\n{\nFOO\n}\n'''"
5,3-5,4:NEWLINE'\n'   
6,0-6,0:ENDMARKER  ''

The f-string could should be tokenized as something like:
FSTRING_START f'''
STRING_PART \n 
LEFT_BRACE {
NEWLINE
IDENTIFIER FOO
NEWLINE
RIGHT_BRACE }
STRING_PART \n
FSTRING_END '''

Although this would complicate the tokenizer, it would mean that the internals 
of f-strings could be made explicit in the grammar, and that the compiler could 
generate correct offsets.

--
nosy: +Mark.Shannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Ivan Levkivskyi]

Changes by Ivan Levkivskyi :


--
nosy: +levkivskyi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
dependencies: +Tracebacks should contain the first line of continuation lines

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[R. David Murray]

R. David Murray added the comment:

These are not problems with f-strings in particular, they are problems in 
general with the way python parsing and error reporting happens.  The second is 
presumably (I haven't gotten around to understanding how f-strings work under 
the hood) an example of error reporting from a separately evaled string.

Improvements in this area are certainly welcome.  There is an open issue 
relevant to your first example, issue 12458.  I'm sure that f-strings 
complicate the solution at least slightly, but I think there are more 
fundamental pre-requisites to be addressed first in solving it.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29051] Improve error reporting involving f-strings (PEP 498)

[Chi Hsuan Yen]

New submission from Chi Hsuan Yen:

Here are the two examples I found confusing when playing with f-strings. The 
first one involves with a NameError:

$ cat test2
f'''
{
FOO
}
'''

$ python3.7m test2
Traceback (most recent call last):
  File "test2", line 5, in 
'''
NameError: name 'FOO' is not defined


It would be better if the error reporter points to the actual line of the error:

$ python3.7m test2
Traceback (most recent call last):
  File "test2", line 3, in 
FOO
NameError: name 'FOO' is not defined

The second one involves a SyntaxError:

$ cat test2 
f'''
{
a b c
}
'''

$ python3.7m test2
  File "", line 2
a b c
  ^
SyntaxError: invalid syntax

It would be better if the line number is relative to the file instead of the 
expression in f-strings:

$ python3.7m test2
  File "test2", line 3
a b c
  ^
SyntaxError: invalid syntax

By the way, external checkers like pyflakes also suffers. They rely on ASTs. 
Nodes in f-strings have their lineno relative to the {...} expression instead 
of the whole code string. For example:

import ast

code = '''
f'{LOL}'
'''

for node in ast.walk(ast.parse(code, "", "exec")):
if isinstance(node, ast.Name):
print(node.lineno)


Prints 1 instead of 2.

Another by the way, ruby reports correct line numbers:

$ cat test3 
"
#{
FOO
}
"

$ ruby test3 
test3:3:in `': uninitialized constant FOO (NameError)


$ cat test3 
"
#{
@@
}
"

$ ruby test3
test3:3: `@@' without identifiers is not allowed as a class variable name
test3:3: syntax error, unexpected end-of-input


Added the author and the primary reviewer of issue24965.

--
components: Interpreter Core
messages: 283874
nosy: Chi Hsuan Yen, eric.smith, martin.panter
priority: normal
severity: normal
status: open
title: Improve error reporting involving f-strings (PEP 498)
type: enhancement
versions: Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com