[issue20703] RuntimeError caused by lazy imports in pdb

[Irit Katriel]

Irit Katriel  added the comment:

Closing as this is not a problem with pdb, it's a general problem with 
iterating sys.modules while doing things that can cause imports.

--
resolution:  -> wont fix
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Irit Katriel]

Irit Katriel  added the comment:

Can we make sys.modules automatically return an iterator that has a copy of its 
contents?  Otherwise it's an odd API - it's iterable but we tell people not to 
iterate it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

My concern about importing readline at the top of the module is that importing 
readline has a side effect. The only module which imports it unconditionally at 
the top is rlcompleter. In all other places it is imported lazily.

And importing readline ahead may not fix the original issue, because there are 
other modules lazily imported in pdb: runpy, shlex, pydoc.

Perhaps we should just say that sys.modules should not be iterated directly, 
you should first make a copy. There are other issues about RuntimeError raised 
during iterating sys.modules.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Irit Katriel]

Irit Katriel  added the comment:

I think option 2 changes the current behaviour, because cmd.Cmd.cmdloop will 
import readline later, and the set_completer_delims() call would never happen 
even though readline is used. I'll update the patch to do option 1.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Having a side effect at import time is not good. It will interfere with 
programs which just import pdb, but not use it.

There are two other options:

1. Import readline at top level, but call set_completer_delims() lazily.

2. Do not import readline at all. Call set_completer_delims() only if readline 
is already imported (sys.modules.get('readline') is not None).

I prefer the latter one.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Irit Katriel]

Irit Katriel  added the comment:

I agree with both Eric and Xavier - any library calling a function that 
modifies sys.modules will see this issue, but pdb should try not to modify 
program semantics like this. 

It's fixed if we move the readline import as Louie suggested (the _bootlocale 
import was probably removed by now, I don't see it).

--
versions: +Python 3.10, Python 3.11, Python 3.9 -Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Irit Katriel]

Change by Irit Katriel :


--
keywords: +patch
nosy: +iritkatriel
nosy_count: 5.0 -> 6.0
pull_requests: +26035
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27520

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Louie Lu]

Changes by Louie Lu :


--
versions: +Python 3.6, Python 3.7 -Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Louie Lu]

Louie Lu added the comment:

The lazy import cause by two modules, readline and _bootlocale.

readline: in __init__

   try:
   import readline
   readline.set_completer_delims(' \t\n`@#$%^&*()=+[{]}\\|;:\'",<>?')

_bootlocale: in __init__

with open(os.path.join(envHome, '.pdbrc')) as rcFile:
with open('.pdbrc') as rcFile

Easiest way to eliminated it to move import readline and import _bootlocale to 
the top of the file.

--
nosy: +louielu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Xavier de Gaye]

Xavier de Gaye added the comment:

 In that case the behavior you are seeing is correct, even if not obvious or 
 even desirable.  It will happen any time you are looping over sys.modules and 
 call a function/method which has a function-scoped import statement for a 
 module that hasn't been imported yet (or calls another function that does so, 
 etc.).

I do not agree. Suppose there is some piece of code that loops over sys.modules 
without doing any lazy import (and thus without raising the RuntimeError 
dictionary changed size during iteration). Now, if you insert a 
pdb.set_trace() in that code, you will get the RuntimeError and this is not 
correct.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20703
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Eric Snow]

Eric Snow added the comment:

I haven't looked to closely but I'm guessing that pdb.set_trace() causes 
something to get imported (i.e. there's an import statement in a function body 
somewhere).  Consequently sys.modules is updated (by that distant import 
statement) while you are iterating over it here.

In that case the behavior you are seeing is correct, even if not obvious or 
even desirable.  It will happen any time you are looping over sys.modules and 
call a function/method which has a function-scoped import statement for a 
module that hasn't been imported yet (or calls another function that does so, 
etc.).

--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20703
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Mark Lawrence]

Mark Lawrence added the comment:

My tests were done with a script from the command line, not an interactive 
prompt.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20703
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Xavier de Gaye]

Xavier de Gaye added the comment:

 I can reproduce this on Windows 8.1 with 3.4.3 but cannot do so with 3.5.0a1.

I can still reproduce this on linux on today's tip:
'3.5.0a1+ (default:7185a35fb293, Feb 26 2015, 11:27:11) \n[GCC 4.9.2 20150204 
(prerelease)]'.

Maybe you tried reproducing it with the interactive interpreter. When this code
is run from the interactive interpreter and not as a script as suggested in the
initial post, then readline is already imported by the interpreter at startup
and the RuntimeError is not raised.

Also interestingly, the bdb module takes lot of steps to ensure that linecache
and reprlib are lazyly imported, but pdb and bdb cannot do anything interesting
without these modules.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20703
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Mark Lawrence]

Mark Lawrence added the comment:

I can reproduce this on Windows 8.1 with 3.4.3 but cannot do so with 3.5.0a1.

--
nosy: +BreamoreBoy
versions: +Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20703
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Josh Rosenberg]

Josh Rosenberg added the comment:

Is this worth fixing? Delve into the innards and you get weird behaviors. If 
you really need to iterate sys.modules (the example clearly doesn't, but that 
doesn't mean no one would), you could just copy the parts you need beforehand 
to get a consistent view, e.g.

for name, mod in tuple(sys.modules.items()):

Lazy module imports aren't always a bad thing if the module in question isn't 
needed for the common functionality, only specific subsets that may never be 
used.

--
nosy: +josh.r

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20703
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20703] RuntimeError caused by lazy imports in pdb

[Xavier de Gaye]

New submission from Xavier de Gaye:

Issuing the 'continue' pdb command with a lazy_import.py script as:

# START of lazy_import.py
import sys, pdb

for m in sys.modules:
if m == 'sys':
pdb.set_trace()

# END of lazy_import.py


gives the following output:

$ python lazy_import.py
 lazy_import.py(3)module()
- for m in sys.modules:
(Pdb) continue
Traceback (most recent call last):
  File lazy_import.py, line 3, in module
for m in sys.modules:
RuntimeError: dictionary changed size during iteration

--
components: Library (Lib)
messages: 211737
nosy: georg.brandl, xdegaye
priority: normal
severity: normal
status: open
title: RuntimeError caused by lazy imports in pdb
type: behavior
versions: Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20703
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com