[issue26452] Wrong line number attributed to comprehension expressions

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Thanks much Greg for the details but I am closing this as fixed since there 
have been some improvements merged with issue12458 along with tests. The 
original issue reported regarding multi-line comprehension was fixed as I have 
tested it on https://bugs.python.org/issue26452#msg325985. I tested the patch 
but unfortunately had runtime errors. Feel free to reopen this if needed or any 
cases have been missed in the patch that you would like to make a PR.

Thanks :)

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26452] Wrong line number attributed to comprehension expressions

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

I think there have been some improvements merged with 
https://bugs.python.org/issue12458 . I am not sure if the patch covers more 
cases. I ran the tests attached in the patch and some of them cause 
RuntimeError in master.

# bpo26452.py

def f():
return [j
for i in range(3)
if i]

f()

# Master 

./python.exe
Python 3.8.0a0 (heads/master:c510c6b8b6, Sep 21 2018, 11:10:24)
[Clang 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

./python.exe ../backups/bpo26452.py
Traceback (most recent call last):
  File "../backups/bpo26452.py", line 6, in 
f()
  File "../backups/bpo26452.py", line 2, in f
return [j
  File "../backups/bpo26452.py", line 2, in 
return [j
NameError: name 'j' is not defined


# Python 3.7

python3.7 ../backups/bpo26452.py
Traceback (most recent call last):
  File "../backups/bpo26452.py", line 6, in 
f()
  File "../backups/bpo26452.py", line 3, in f
for i in range(3)
  File "../backups/bpo26452.py", line 4, in 
if i]
NameError: name 'j' is not defined

# Tests

./python.exe ../backups/test26452.py
..F
==
FAIL: test_operators (__main__.TestPEP380Operation)
--
Traceback (most recent call last):
  File "../backups/test26452.py", line 21, in expect_line
f()
  File "../backups/test26452.py", line 98, in f
-
RuntimeError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../backups/test26452.py", line 100, in test_operators
self.expect_line(f, 3) # want 2
  File "../backups/test26452.py", line 26, in expect_line
self.assertEqual(relative_line, expected_relative_line)
AssertionError: 2 != 3

--
Ran 3 tests in 0.001s

FAILED (failures=1)


Thanks

--
nosy: +xtreak

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26452] Wrong line number attributed to comprehension expressions

[SilentGhost]

Changes by SilentGhost :


--
nosy: +haypo, serhiy.storchaka
versions:  -Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26452] Wrong line number attributed to comprehension expressions

[Greg Price]

New submission from Greg Price:

In a multi-line list comprehension (or dict or set comprehension), the code for 
the main expression of the comprehension is wrongly attributed to the *last* 
line of the comprehension, which might be several lines later.

This makes for quite baffling tracebacks when an exception occurs -- for 
example this program:
```
def f():
return [j
for i in range(3)
if i]

f()
```
produces (with CPython from current `default`):
```
Traceback (most recent call last):
  File "foo.py", line 15, in 
f()
  File "foo.py", line 3, in f
for i in range(3)
  File "foo.py", line 4, in 
if i]
NameError: name 'j' is not defined
```
showing the line `if i]`, which has nothing to do with the error and gives very 
little hint as to where the exception is being raised.

Disassembly confirms that the line numbers on the code object are wrong:
```
  2   0 BUILD_LIST   0
  3 LOAD_FAST0 (.0)
>>6 FOR_ITER18 (to 27)

  3   9 STORE_FAST   1 (i)

  4  12 LOAD_FAST1 (i)
 15 POP_JUMP_IF_FALSE6
 18 LOAD_GLOBAL  0 (j)
 21 LIST_APPEND  2
 24 JUMP_ABSOLUTE6
>>   27 RETURN_VALUE
```
The `LOAD_GLOBAL` instruction for `j` is attributed to line 4, when it should 
be line 2.

A similar issue affects multi-line function calls, which get attributed to a 
line in the last argument.  This is less often so seriously confusing because 
the function called is right there as the next frame down on the stack, but 
it's much more common and it makes the traceback look a little off -- I've 
noticed this as a minor annoyance for years, before the more serious 
comprehension issue got my attention.

Historically, line numbers were constrained to be wrong in these ways because 
the line-number table `co_lnotab` on a code object required its line numbers to 
increase monotonically -- and the code for the main expression of a 
comprehension comes after all the `for` and `if` clauses, so it can't get a 
line number earlier than theirs.  Victor Stinner's recent work in 
https://hg.python.org/cpython/rev/775b74e0e103 lifted that restriction in the 
`co_lnotab` data structure, so it's now just a matter of actually entering the 
correct line numbers there.

I have a draft patch to do this, attached here.  It fixes the issue both for 
comprehensions and function calls, and includes tests.  Things I'd still like 
to do before considering the patch ready:
* There are a couple of bits of logic that I knock out that can probably be 
simplified further.
* While I'm looking at this, there are several other forms of expression and 
statement that have or probably have similar issues, and I'll want to go and 
review them too to either fix or determine that they're fine.  The ones I've 
thought of are included in the draft test file, either as actual tests (with 
their current answers) or TODO comments for me to investigate.

Comments very welcome on the issue and my draft patch, and meanwhile I'll 
continue with the further steps mentioned above.

Thanks to Benjamin Peterson for helping diagnose this issue with me when we ran 
into a confusing traceback that ran through a comprehension.

--
components: Interpreter Core
files: lines.diff
keywords: patch
messages: 260966
nosy: Greg Price
priority: normal
severity: normal
status: open
title: Wrong line number attributed to comprehension expressions
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file42042/lines.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com