[issue30403] PEP 547: Running extension modules using -m switch

[Stefan Behnel]

Stefan Behnel added the comment:

I'm a bit torn on this. On the one hand, it's basically saying, "Cython is 
probably going to do it right anyway, so let's just assume it does". That's 
nice, and might be applicable to other cases as well. But that also feels like 
it could need some kind of versioning.

On the other hand, it's totally not magic to implement something similar by 
hand, so naming the flag in a Cython specific way feels wrong from a design 
perspective. Other tools might start picking it up, and that would lead to 
major confusion. In a way, it's both very broad and too narrow.

Basically, if we expect the flag to be used in a broader way, I'm happy to 
generally mark Cython modules with it. It's very explicit in *that* regard. I'm 
just not sure that the use case at hand is the right reason to introduce this 
kind of general marker.

Speaking of versioning, though, what about introducing a generic slot field 
instead that notes the latest CPython API version known to work with the 
module? (Cast pointer value to int to get the value.) That way, CPython could 
introduce new extension module behaviour with new C-API versions, and tools 
that support them can update their version value in the slot to mark them as 
safely supported.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Marcel Plch]

Marcel Plch added the comment:

I have made a patch both for cython and cpython implementing a way to use 
Py_mod_create in cython.

Basically module def that specifies a new "Py_mod_cython" slot are excluded 
from the rule of no module creation, so these modules can be executed directly 
even though they specify Py_mod_create.

Is this approach safe or does it make easy for things to go wrong?

cpython - 
https://github.com/Traceur759/cpython/commit/51a7508d176b23dcf3547b970cf7e6a50898aae2

cython - 
https://github.com/Traceur759/cython/commit/2ca706e10f469cd38947eecd8b92c142532b20bc

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Petr Viktorin]

Petr Viktorin added the comment:

Alright, that makes sense. Thanks for the feedback!
Please give us some time for an updated proposal/implementation. I'm going on 
vacation, so expect about a week.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Stefan Behnel]

Stefan Behnel added the comment:

I was kinda guessing that modifying the slot list wasn't a good idea. ;)

My current use case is that I implement the "create" slot because it makes it 
very easy to intercept the spec and its configuration. It is not passed into 
"exec" as such, but I need it to initialise the module namespace with 
"__file__", "__path__", etc.

There is also still the idea of defining our own module type in Cython in order 
to have a place where we can keep C level module globals, and also to support 
module properties. PEP 549 will not be available in older Python versions, even 
if it gets accepted.

Having to choose between main-exec support and these two features seems wrong.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Petr Viktorin]

Petr Viktorin added the comment:

Again, what is the use case? That's a real question, I'm not saying it to 
dismiss your ideas or points of view. It would be very much easier to think 
about a concrete use case, rather than making a general system for the sake of 
how easy it is implementation-wise. (The implementation might be easier now, 
but it might change, and there's a cost to keeping the generality in mind when 
designing things on top of all this.)

Something like the slot you mention can always be added later if it's needed. 
Is it needed now?


Also, the PyModuleDef should never be modified (beyond the one-time 
initialization that sets ob_type -- that's a workaround for not being always 
able to declare the type statically).
It should be possible to make additional, independent module instances from a 
PyModuleDef.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Stefan Behnel]

Stefan Behnel added the comment:

Marcel proposed to disallow main-execution if the extension *might* return 
anything but a real object (not only if it actually does), but that seems 
excessive to me. The actual problem is that we consider it unsafe if the module 
is executed more than once, because it might overwrite module state. But that's 
entirely up to the extension implementation and independent of what it uses as 
module type.

Given how easy it is so create and/or depend on global state in C, I would 
assume that extensions have to be explicitly designed in order to be 
re-executable. Can't we just have another slot that explicitly marks the module 
as such?

What do you think of this protocol:

Before running the exec or main-exec function, the runner checks for a slot 
entry "Py_mod_allow_reexec" (can have value NULL). If not found, it sets the 
function pointers in the exec *and* main-exec slots to NULL to prevent any 
further (or concurrent) re-execution. If the slot function is not NULL on the 
next execution request, it can be called (again).

That effectively prevents any re-execution by default and provides an opt-in 
way for the module to allow it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Petr Viktorin]

Petr Viktorin added the comment:

Do we have a use case for this?
I'd rather avoid making it easy to do the wrong thing, unless it's needed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Stefan Behnel]

Stefan Behnel added the comment:

OTOH, if the created "module" is not a module object, then we could argue that 
the extension implementation is on its own with that case, and has to do its 
own re-execution safety checks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Marcel Plch]

Marcel Plch added the comment:

Sorry for not responding for so long, I didn't work on Python through the 
summer because of some other matters.

Re-execution of the module is not possible, because there is a check on the C 
level, If you call exec_in_module() on an already initialized module, it'll 
raise ImportError.

Also, because of this check, there is the restriction for py_mod_create. 
"Modules" defining this slot might not be module objects at all, so they might 
not have the module state pointer (which is used to flag if the module was 
initialized).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Stefan Behnel]

Stefan Behnel added the comment:

FYI, I've finally managed to find the time for implementing PEP 489 style 
module initialisation in Cython. It was so easy that I'm sorry it took me so 
long to get started. Cython 0.26 is fresh out, so the feature should go into 
0.27.
https://github.com/cython/cython/pull/1794

Note that I ended up implementing the Py_mod_create function which is now 
copying attributes from the spec right after creation. While not strictly 
required, I guess, it turned out to be easiest, but it seems like this does now 
conflict with the current "-m" implementation. I asked on the github commit why 
that restriction exists.

One thing I stumbled over: exec_in_module() is a Python level function. Would 
that make it possible to re-execute an already imported module? That could be 
dangerous, because C level module code often initialises resources that a 
simple re-execution of the module exec function would overwrite without 
cleaning up the old state. I did not (yet?) implement support for m_clear() 
etc., and that might actually turn out to be really risky when it comes to 
supporting arbitrary user code.

OTOH, that case is easy to detect because the module is already completely 
initialised at that point.

As far as I understand it, all that this PEP really changes from the POV of the 
extension module is that it calls the exec function with a different module 
name ("__main__"). Cython already provides that feature itself (by embedding 
CPython in a C main function), so this should be easy to support.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30403] PEP 547: Running extension modules using -m switch

[Nick Coghlan]

Nick Coghlan added the comment:

This proposal is now PEP 547: 
https://github.com/python/peps/commit/cd84e206f58cf929eea235fb894cff1db2a1dabf

--
title: Running extension modules using -m switch -> PEP 547: Running extension 
modules using -m switch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com