[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Change by Terry J. Reedy :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[miss-islington]

miss-islington  added the comment:


New changeset 799f8489d418b7f9207d333eac38214931bd7dcc by Miss Islington (bot) 
in branch '3.9':
bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
https://github.com/python/cpython/commit/799f8489d418b7f9207d333eac38214931bd7dcc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[miss-islington]

miss-islington  added the comment:


New changeset 5ded7efa6a7a232dd4a41e6e65e4dae47146514b by Miss Islington (bot) 
in branch '3.8':
bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
https://github.com/python/cpython/commit/5ded7efa6a7a232dd4a41e6e65e4dae47146514b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[miss-islington]

Change by miss-islington :


--
pull_requests: +23013
pull_request: https://github.com/python/cpython/pull/24185

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 5.0 -> 6.0
pull_requests: +23012
stage: commit review -> patch review
pull_request: https://github.com/python/cpython/pull/24184

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Terry J. Reedy  added the comment:


New changeset 81f87bbf9f65702062021a78abd9b8f82c98a414 by Terry Jan Reedy in 
branch 'master':
bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
https://github.com/python/cpython/commit/81f87bbf9f65702062021a78abd9b8f82c98a414


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I am reluctant to make changes without adding tests, and until I focused again 
on how to do so, because of your ping, I had no idea how to do so.

The added test results in the same error, "AttributeError: 'BinData' object has 
no attribute 'length'" without the patch and passes with it.

--
stage: patch review -> commit review
versions: +Python 3.10 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Change by Terry J. Reedy :


--
keywords: +patch
pull_requests: +23011
stage: test needed -> patch review
pull_request: https://github.com/python/cpython/pull/24183

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Timothy Geiser]

Timothy Geiser  added the comment:

This issue is still open, 8.5 months after identifying the underlying cause.
What needs done to get a fix for this merged? Manual testing, or adding test 
cases? Anything I can do to help? I have 3.9.1 successfully built on a 
Raspberry Pi that I can reproduce the issue on, as well as reproduce the fix 
Terry suggested in msg366981.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Terry J. Reedy  added the comment:

SO duplicate.
https://stackoverflow.com/questions/61310989/python-idle-importing-xlrd-error-generated-in-debug-mode-attributeerror-mo

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Timothy Geiser]

Timothy Geiser  added the comment:

Looks good when testing both the minimal example and the pgpdump original case.

I added the import at the top, and changed the original line 173 from

value = repr(value)

to

value = reprlib.repr(value)

This is based on lines 160 & 161 in reprlib (we need a Repr instance, but 
reprlib makes it's own for us to use).

I get a nice boring and not-crashy result in the debug window for Locals:

data   b''
self   

Thanks for the fix, Terry!
I shouldn't expect any strange side-effects from this, should I?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[ppperry]

Change by ppperry :


--
nosy:  -ppperry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Timothy, can you try editing idlelib.debugger_r, line 173, as suggested above 
and see if it solves the problem?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I believe I found the bug.  For IDLE's original single process mode, still 
available with the -n startup option, debugger.py contains the entire debugger. 
 The values displayed for global and local names are obtained with 
reprlib.Repr.repr.  That in turn calls .repr1, and that calls .repr_xxx, where 
xxx is one of the 'common' builtin classes or 'instance'.

The latter is used for all user classes.  It calls __builtins__.repr, but 
guarded by 'try...except Exception' since user classes may cause exceptions.  
The except clause returns an alternative type and id string, like 
object.__repr__.  (That alternative could also raise, but much less often.  Any 
of the examples above should run if IDLE were started from a command line with 
'python -m idlelib -n'.

When user code is run in a separate process, the code that interacts with user 
object must also run in the separate process.  debugger.Idb is moved and the 
code in debugger_r is added, some in each process.  Of concern here is that the 
GUI code that displays global or local values is passed a dict proxy instead of 
an actual namespace dict.  The proxy __getitem__ for d[key] makes an rpc call 
to through the socket connection to code in the user process.  That returns not 
the object itself but a string representation.  It does so with an unguarded 
repr call.

IDLE intentionally removes traceback lines added by IDLE (and pdb, if used), so 
that tracebacks look mostly the same as in standard CPython.  But that is a 
handicap when there is a bug in IDLE.  A traceback ending with 
  File .../idlelib/debugger_r, line 173, in dict_item
value = repr(value)
AttributeError: ...

would have been a big help here.  I am thinking about how to selectively 
disable traceback cleanup.

In any case, I believe the solution is to import reprlib in debugger_r also and 
add 'reprlib.Repr' before 'repr' in the line above.

--
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Timothy Geiser]

Timothy Geiser  added the comment:

It looks like the IDLE debugger seems to call repr on objects to present in the 
Locals list, even before they've been properly initialized. If __repr__ needs 
to refer to variables that don't exist until __init__ is done (and I don't 
think it's unreasonable for __repr__ to assume that __init__ is indeed 
finished), the debugger either needs wait until __init__ has completed on any 
given instance before trying to repr it, or otherwise needs to catch a 
potentially very wide range of exceptions that might be raised from calling 
__repr__ so early. I prefer the latter solution, since any buggy code that 
(effectively) crashes on it's __repr__ (for whatever reason) will probably 
bring the debugger to it's knees.
I played around with pdb directly and can sort of get the same thing if I ask 
for __repr__ too early:

 1 Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit 
(AMD64)] on win32
 2 Type "help", "copyright", "credits" or "license" for more information.
 3 >>> import data
 4 >>> import pdb
 5 >>> pdb.run("data.BinaryData(b'some data')")
 6 > (1)()
 7 (Pdb) step
 8 --Call--
 9 > c:\users\geitd\documents\python\data.py(4)__init__()
10 -> def __init__(self, data):
11 (Pdb) p self
12 (Pdb) p BinaryData.__repr__(self)
13 *** AttributeError: 'BinaryData' object has no attribute 'length'
14 (Pdb) step
15 > c:\users\geitd\documents\python\data.py(5)__init__()
16 -> if not data:
17 (Pdb) step
18 > c:\users\geitd\documents\python\data.py(7)__init__()
19 -> if len(data) <= 1:
20 (Pdb) step
21 > c:\users\geitd\documents\python\data.py(10)__init__()
22 -> self.data = data
23 (Pdb) step
24 > c:\users\geitd\documents\python\data.py(11)__init__()
25 -> self.length = len(data)
26 (Pdb) p self
27 (Pdb) p BinaryData.__repr__(self)
28 *** AttributeError: 'BinaryData' object has no attribute 'length'
29 (Pdb) step
30 --Return--
31 > c:\users\geitd\documents\python\data.py(11)__init__()->None
32 -> self.length = len(data)
33 (Pdb) p self
34 
35 (Pdb) p BinaryData.__repr__(self)
36 ''

Note that line 11 didn't return anything, but didn't have any bad results, 
whereas the way I phrased line 12 gave the exact same error the IDLE debugger 
threw.
Lines 26 and 27 towards the end of __init__ came out the same, but after the 
--Return-- on 30, either phrasing gives what you'd expect.
I suppose the TL;DR is to take the mechanism that gives the correct behavior of 
'p self' in pdb and copy it over to the IDLE debugger (or whatever other 
mechanism is necessary).

Is this enough for you to work from?

--
Added file: https://bugs.python.org/file49082/data.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Timothy, this is already more helpful than a simple 'me too'.  Can you reduce 
pgdata.dump to a truly minimal file that exhibits the bug?  and then upload it? 
 Installing any package into my local copy of the master CPython repository is 
problemmatical.  What I need is a file that I can download elsewhere, load into 
IDLE, run from a branch for this issue, and edit and rerun.  From the 
traceback, it appears that it might be as simple as 

class BinaryData:
def __init__(self, bd):


BinaryData(b'')

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Timothy Geiser]

Timothy Geiser  added the comment:

I wish I could be more helpful than to just pipe up with a "this bug affects 
me, too," note, but wanted to poke this bug report since it's been dormant for 
14 months.
With Python 3.8.2 I tried using the pgpdump module (version 1.5, installed from 
pip) on Windows 10 and wanted to step through how it worked. As soon as I 
enabled the debugger in IDLE it stopped working, throwing a very similar stack 
trace. Here's the MWE:

>>> import pgpdump
>>> 
[DEBUG ON]
>>> pgpdump.BinaryData(b'')


Traceback (most recent call last):
  File "", line 1, in 
pgpdump.BinaryData(b'')
  File "C:\Python38\lib\site-packages\pgpdump\data.py", line 13, in __init__
if not data:
  File "C:\Python38\lib\site-packages\pgpdump\data.py", line 13, in __init__
if not data:
  File "C:\Python38\lib\bdb.py", line 88, in trace_dispatch
return self.dispatch_line(frame)
  File "C:\Python38\lib\bdb.py", line 112, in dispatch_line
self.user_line(frame)
  File "C:\Python38\lib\idlelib\debugger.py", line 24, in user_line
self.gui.interaction(message, frame)
AttributeError: 'BinaryData' object has no attribute 'length'

This error only occurs when the Locals checkbox in the debugging window is 
checked - it runs as expected as long as Locals is unchecked (this minimum 
[not]working example throws a "pgpdump.utils.PgpdumpException: no data to 
parse" error, as it should). A longer example with actual data will run for 
several steps while Locals is unchecked, but fails with the first Step once 
Locals is checked. A side note is that the specific error here is that the 
class BinaryData is being asked about it's 'length', rather than the OP's error 
of _ModuleLock not having a 'name'
Is there anything else I can do to help fix this, given that I'm not familiar 
with programming bdb or the IDLE debugger GUI?

--
nosy: +geitda

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Terry J. Reedy  added the comment:

A StackOverflow use got the same _ModuleLock message.
https://stackoverflow.com/questions/54785596/debugger-in-python-freezes-over-own-built-modules

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Change by Terry J. Reedy :


--
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[ppperry]

ppperry  added the comment:

Line 59 isn't actually executed; the error comes from the trace event that gets 
fired before line 59, which is the first `line` event in the frame containing 
the uninitialized _ModuleLock.

--
nosy: +ppperry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33065] IDLE debugger: failure stepping through module loading

[Terry J. Reedy]

Terry J. Reedy  added the comment:

The problem is not limited to user modules.  In duplicate issue #34609 (not 
closed), the same traceback was obtained with unittest.  I also got the same 
with asyncio.  There will not be a problem if the module is already loaded in 
sys.modules, so that importlib is not invoked.

The traceback is obscured by the fact that the executed importlib is frozen, 
leaving it traceable, but the code not directly available.  Hence the code is 
omitted from the debugger display and traceback.  However, the line numbers can 
be used to find the code within Lib/importlib._bootstrap.py.  Using current 
master (updated last night), the functions and lines executed by stepping with 
import unittest are (as expected when reading the code, up to the unexpected 
exception):

_find_and_load: 980
ModuleLock.__init__: 144, 145
ModuleLock.__enter__: 148
_get_module_lock: 163-168, 170-171, 174: lock = _ModuleLock(name)
_ModuleLock.__init__: 59: self.lock = _thread.allocate_lock()

IDLE's visual debugger has name-value panels for locals, including non-locals, 
and for globals.  It uses repr to gets value representations.  The locals panel 
is displayed by default.

Before line 174, 'lock' is bound to None, so before executing line 59, the 
display is "lock:None\nname:'unittest'".  After line 59, debugger apparently 
tries to get the repr of the in-process instance.  Since the call in 174 has 
not completed and should not have rebound 'lock' to the instance, I do not 
understand why.  (Brett, I now understand what you wrote to be pointing as this 
puzzle also.)  ppperry, additional light would be appreciated.

Given that debugger does try to get the repr of the instance, both Brett 
Cannon, here, and (ppperry), on duplicate issue #34609 (now closed), have 
pointed out that _ModuleLock.__repr__ uses self.name:
return '_ModuleLock({!r}) at {}'.format(self.name, id(self))

I verified that updating the locals panel is the problem by starting over and 
turning the panel off until past the the assignment to self.name, at which 
point, the lock value is "_ModuleLock('unittest') at ...".

Debugger should be prepared for repr to fail, and display something 
informative.  In the present case, perhaps

repr raised "AttributeError: '_ModuleLock' object has no attribute 'name'"

with a check for whether the exception message contains "'.*' object" (and add 
such if not present).

--
nosy:  -ppperry
title: IDLE debugger crashes when `repr` raises an exception -> IDLE debugger: 
failure stepping through module loading
versions: +Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com