[issue23642] Interaction of ModuleSpec and C Extension Modules

[Petr Viktorin]

Change by Petr Viktorin :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[Petr Viktorin]

Petr Viktorin added the comment:

ping; this issue can be closed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[Petr Viktorin]

Petr Viktorin added the comment:

Since PEP 489 (Python 3.5.0b1), loaders now support create_module/exec_module. 
For modules which don't use the PEP's new multi-phase init mechanism (which is 
most of them, currently), load_module() does all the work and exec_module is a 
no-op.

So, this issue can be closed.




Python 3.5.0b2+ (default, Jun  2 2015, 20:48:59)
   
[GCC 4.9.2 20150212 (Red Hat 4.9.2-6)] on linux 
   
Type help, copyright, credits or license for more information.  
   
 from importlib.util import find_spec, module_from_spec  

 spec = find_spec('math')

 spec

ModuleSpec(name='math', loader=_frozen_importlib_external.ExtensionFileLoader 
object at 0x7f7d67b51588, 
origin='/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so')
   
 math = module_from_spec(spec)   

 math

module 'math' from 
'/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so'
 
 dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 
'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 
'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 
'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 
'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 
'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 
'tan', 'tanh', 'trunc']
 spec.loader.exec_module(math)
 spec.loader.load_module(spec.name)
module 'math' from 
'/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so'
 dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 
'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 
'cos', 'cosh', 'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 
'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'gcd', 'hypot', 'inf', 
'isclose', 'isfinite', 'isinf', 'isnan', 'ldexp', 'lgamma', 'log', 'log10', 
'log1p', 'log2', 'modf', 'nan', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 
'tan', 'tanh', 'trunc']

The array module implements multi-phase init, so create_module/exec_module 
work more like with Python modules:

 spec = find_spec('array')
 spec
ModuleSpec(name='array', loader=_frozen_importlib_external.ExtensionFileLoader 
object at 0x7f7d67b51f60, 
origin='/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/array.cpython-35m-x86_64-linux-gnu.so')
 array = module_from_spec(spec)
 array
module 'array' from 
'/home/petr/dev/cpython/__installed__/lib/python3.5/lib-dynload/array.cpython-35m-x86_64-linux-gnu.so'
 dir(array)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 
'_array_reconstructor']
 spec.loader.exec_module(array)
 dir(array)
['ArrayType', '__doc__', '__file__', '__loader__', '__name__', '__package__', 
'__spec__', '_array_reconstructor', 'array', 'typecodes']

(Module methods [like _array_reconstructor here] and __doc__ are initialized in 
load_module -- an implementation detail related to reusing the existing C 
structure for defining extension modules.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[Raúl Cumplido]

Changes by Raúl Cumplido raulcumpl...@gmail.com:


--
nosy: +raulcd

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[Stefan Behnel]

Changes by Stefan Behnel sco...@users.sourceforge.net:


--
nosy: +scoder

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[Nick Coghlan]

Nick Coghlan added the comment:

Added Petr to the nosy list here as well.

Petr - this is the kind of discrepancy I'm hoping that PEP 489 can help remedy, 
so it may make for a good test case :)

--
nosy: +encukou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[Brett Cannon]

Brett Cannon added the comment:

The PEP is now live: https://www.python.org/dev/peps/pep-0489/ . If you want to 
participate in the discussion it will probably happen on the import-sig.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[David Beazley]

David Beazley added the comment:

Note: Might be related to Issue 19713.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[Brett Cannon]

Brett Cannon added the comment:

There is a proposed PEP on the import-sig: 
https://mail.python.org/pipermail/import-sig/2015-March/000904.html . I'm 
hoping to add it for the author to the PEP index on Friday.

Basically we punted on extension modules as we ran out of time in Python 3.4.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[David Beazley]

David Beazley added the comment:

This is great news.   Read the PEP draft and think this is a very good thing to 
be addressing. Thanks, Brett.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[David Beazley]

David Beazley added the comment:

inal comment.  It seems that one can generally avoid a lot of nastiness if 
importlib.reload() is used instead.  For example:

 mod = sys.modules[spec.name] = module_from_spec(spec)
 importlib.reload(mod)

This works for both source and Extension modules and completely avoids the need 
to worry about the exec_module()/load_module() warts.   Wouldn't say it's an 
obvious approach though ;-).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[Berker Peksag]

Changes by Berker Peksag berker.pek...@gmail.com:


--
nosy: +brett.cannon, eric.snow, ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[David Beazley]

David Beazley added the comment:

Sorry. I take back the previous message.  It still doesn't quite do what I 
want.   Anyways, any insight or thoughts about this would be appreciated ;-).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23642] Interaction of ModuleSpec and C Extension Modules

[David Beazley]

New submission from David Beazley:

I have been investigating some of the new importlib machinery and the addition 
of ModuleSpec objects.  I am a little curious about the intended handling of C 
Extension modules going forward. 

Backing up for a moment, consider a pure Python module.  It seems that I can do 
things like this to bring a module into existence (some steps involving 
sys.modules omitted).

 from importlib.util import find_spec, module_from_spec
 spec = find_spec('socket')
 socket = module_from_spec(spec)
 spec.loader.exec_module(socket)


However, it all gets weird with C extension modules.  For example, you can 
perform the first few steps:

 spec = find_spec('math')
 spec
ModuleSpec(name='math', loader=_frozen_importlib.ExtensionFileLoader object at 
0x1012122b0, origin='/usr/local/lib/python3.5/lib-dynload/math.so')
 math = module_from_spec(spec)
 math
module 'math' from '/usr/local/lib/python3.5/lib-dynload/math.so'
 dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']

As you can see, you get a fresh unloaded module here.  However, if you try to 
bring in the module contents, things get screwy.

 spec.loader.exec_module(math)
Traceback (most recent call last):
  File stdin, line 1, in module
AttributeError: 'ExtensionFileLoader' object has no attribute 'exec_module'


Yes, this is the old legacy interface in action--there is no exec_module() 
method.   You can always fall back to load_module() like this:

 spec.loader.load_module(spec.name)
module 'math' from '/usr/local/lib/python3.5/lib-dynload/math.so'


The problem here is that it creates a brand new module and ignores the one that 
was previously created by module_from_spec().  That module is still empty:

 dir(math)
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']
 

I realize that I'm treading into a swamp of legacy interfaces and some pretty 
complex machinery here.  However, here's my question:  are C extension modules 
always going to be a special case that need to be considered code that 
interacts with the import system.  Specifically, will it need to be 
special-cased to use load_module() instead of the 
module_from_spec()/exec_module() combination?

I suppose the question might also apply to built-in and frozen modules as well 
(although I haven't investigated that so much). 

Mainly, I'm just trying to gain some insight from the devs as to the overall 
direction where the import implementation is going with this.

P.S.  ModuleSpecs are cool. +1

--
components: Interpreter Core
messages: 237872
nosy: dabeaz
priority: normal
severity: normal
status: open
title: Interaction of ModuleSpec and C Extension Modules
type: behavior
versions: Python 3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23642
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com