[issue12633] sys.modules gets special treatment

[Nick Coghlan]

Nick Coghlan  added the comment:

+1 for making this limitation explicit. See the caveat on locals() [1] for an 
example of how to note this kind of restriction.

[1] http://docs.python.org/dev/library/functions.html#locals

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12633] sys.modules gets special treatment

[Eric Snow]

New submission from Eric Snow :

The sys.modules dict is a special object.  It is the only variable of the 
CPython interpreter object that is exposed in the sys module[1].  Everything 
else in sys lives in the module.  However, the modules dict lives in the 
interpreter object and is bound to the sys module separately.  No other 
variable of the interpreter object gets this treatment.

This situation sets up an unexpected behavior for sys.modules.  There are many 
places, mostly in Python/import.c, where the modules dict gets used and not by 
pulling from sys.modules.  These places use interp->modules directly[2].  So if 
sys.modules is re-bound, the imp module is using/reporting an out of sync 
modules dict.

One could argue that re-binding a module global is risky and should be avoided. 
 I agree.  Here is the use case that prompted me to march ahead anyway:

class BaseTest(TestCase):
@classmethod
def setUpClass(cls):
cls.sysmodules = sys.modules
sys.modules = sys.modules.copy()
@classmethod
def tearDownClass(cls):
sys.modules = cls.sysmodules

I was writing some import related tests and wanted sys.modules to be returned 
to its initial state after each test.  I realise that Lib/test/support.py 
provides CleanImport and others address this, but you have to provide the 
module names to clean up.  This is an unfortunate hassle sometimes when several 
layers of imports happen during the import of the module you care about.

So the result was an exception when I tried importing an extension module, like 
"_sqlite3".  This is because in importdl.h the new module is added to the dict 
returned by PyImport_GetModuleDict(), not to the one at sys.modules.

For now I am doing the following to get the same effect:

class BaseTest(TestCase):
@classmethod
def setUpClass(cls):
cls.sysmodules = sys.modules.copy()
@classmethod
def tearDownClass(cls):
for name in sys.modules:
del sys.modules[name]
for name in cls.sysmodules:
sys.modules[name] = cls.sysmodules[name]

However, this is less efficient, sort of.  I expect that the current direct use 
of interp->modules in the CPython code is [much?] more efficient than 
PySys_GetObject("modules") calls.

Proposal

In light of all this I recommend that either use of interp->modules be replaced 
by PySys_GetObject("modules") calls, or the sys module documentation[3] be 
updated to make clear that sys.modules should not be re-bound (in a CPython 
implementation detail note).  I'm guessing that the first option is right out.  
The documentation addition would be just right.


[1] variables of the interpreter object found by grepping "interp->" in the 
CPython source:
 modules
 modules_by_index
 next
 codec_search_path
 codec_search_cache
 codec_error_registry
 codecs_initialized
 fscodec_initialized
 modules_reloading
 builtins
 sysdict
 tstate_head
 tscdump
 dlopenflags
[2] see PyImport_GetModuleDict() in Python/import.c
[3] http://docs.python.org/dev/library/sys.html#sys.modules

--
messages: 141068
nosy: ericsnow, ncoghlan
priority: normal
severity: normal
status: open
title: sys.modules gets special treatment
type: behavior
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com