Re: [Python-Dev] PEP 395: Module Aliasing
Just now I was trapped by the "Importing the main module twice" problem for a while, and then I searched on web and go into here. It seems pretty good to fix this problem, since it is really difficult to find out this problem. Anyway, a module should not be initialized twice which results in that each of the objects defined in this module has TWO instances in separate modules. Accidentally people will mix up them together. -- Ray Allen Best wishes! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On 09/03/2011 16:30, P.J. Eby wrote: At 05:35 PM 3/4/2011 +, Michael Foord wrote: That (below) is not distutils it is setuptools. distutils just uses `scripts=[...]`, which annoyingly *doesn't* work with setuptools. Er, what? That's news to me. Could you file a bug report about what doesn't work, specifically? Ok. I'm afraid I was relying on anecdotal evidence (the people who wanted setuptools specific features in my setup.py told me I *had* to use entry points for scripts instead of the standard distutils scripts machinery). I'll try out a vanilla "scripts" entry in a setuptools setup.py and report back. All the best, Michael -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
At 05:35 PM 3/4/2011 +, Michael Foord wrote: That (below) is not distutils it is setuptools. distutils just uses `scripts=[...]`, which annoyingly *doesn't* work with setuptools. Er, what? That's news to me. Could you file a bug report about what doesn't work, specifically? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Sun, Mar 6, 2011 at 2:54 PM, Ron Adam wrote: > > ... if the key matches sys.modules.__missing__ > > Works for me. ;-) > > We can find a better name than __missing__ later. (minor detail) __missing__ is a dict method. I was merely noting it would make more sense to override that rather than __setitem__ to avoid slowing down normal lookups in sys.modules. However, I'll still advocate the simple approach (two entries) for now. Assuming uniqueness of values in a dictionary is invalid in general, so code assuming it for sys.modules is arguably already incorrect. Definitely a point deserving of more discussion in the PEP, though. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On 03/05/2011 06:33 PM, Nick Coghlan wrote: On Sun, Mar 6, 2011 at 4:11 AM, Ron Adam wrote: Adding a second references to the '__main__' module begins to blur the purpose of sys.modules from being a place to keep imported modules to a place that also has some ordering information. (Practicality over purity?, Or an indication of what direction to go in?) And, if you ask for keys, items, or values, you will need to filter out '__main__' to avoid the double reference. So I was thinking, what if we make sys.modules a little smarter. The negative of that is, it would no longer be a simple dictionary object. First idea ... Add a __main__ attribute to sys.modules to hold the name of the main module. Override modules.__setitem__, so it will catch '__main__' and set modules.__main__ to the name of the module, and put the module in under its real name instead of '__main__'. Override modules.__getitem__, so it will catch '__main__' and return self[self.__main__]. Then keys(), values(), and items(), will not have the doubled main module references in them. The name of the main module is easy to get. ie... sys.modules.__main__ sys.modules[__name__] will still return the correct module if __name__ == "__main__". That's quite an interesting idea - I hadn't even considered the implications of double counting the affected module when iterating over sys.modules in some fashion. That said, dropping `__main__` from the iteration might confuse existing code, so it may be better to have the lookup go the other way (i.e. define __missing__ on the dict subclass and return sys.modules['__main__'] if the key matches sys.modules.__main__). ... if the key matches sys.modules.__missing__ Works for me. ;-) We can find a better name than __missing__ later. (minor detail) Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Sun, Mar 6, 2011 at 4:11 AM, Ron Adam wrote: > Adding a second references to the '__main__' module begins to blur the > purpose of sys.modules from being a place to keep imported modules to a > place that also has some ordering information. (Practicality over purity?, > Or an indication of what direction to go in?) > > And, if you ask for keys, items, or values, you will need to filter out > '__main__' to avoid the double reference. > > So I was thinking, what if we make sys.modules a little smarter. The > negative of that is, it would no longer be a simple dictionary object. > > First idea ... > > Add a __main__ attribute to sys.modules to hold the name of the main module. > > Override modules.__setitem__, so it will catch '__main__' and set > modules.__main__ to the name of the module, and put the module in under its > real name instead of '__main__'. > > Override modules.__getitem__, so it will catch '__main__' and return > self[self.__main__]. > > Then keys(), values(), and items(), will not have the doubled main module > references in them. > > The name of the main module is easy to get. ie... sys.modules.__main__ > > sys.modules[__name__] will still return the correct module if __name__ == > "__main__". That's quite an interesting idea - I hadn't even considered the implications of double counting the affected module when iterating over sys.modules in some fashion. That said, dropping `__main__` from the iteration might confuse existing code, so it may be better to have the lookup go the other way (i.e. define __missing__ on the dict subclass and return sys.modules['__main__'] if the key matches sys.modules.__main__). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On 03/05/2011 01:14 AM, Nick Coghlan wrote: On Sat, Mar 5, 2011 at 2:40 PM, Ron Adam wrote: Fixing direct execution inside packages +1 on both of these. I don't see any major problems with these. Any minor issues can be worked out. I suggest separating these two items out into their own PEP, and doing them first. This first draft includes everything I *want* to do. If I have to scale the proposal back to get it accepted because I can't convince enough other people it's a good idea... well, even Guido isn't immune to that part of the PEP process (cf. PEP 340). It wouldn't be the first PEP to have a "Rejected Ideas" section, and I'm sure it wouldn't be the last. Right, All of the reasons are good and some solution should be implemented for each of the issues you pointed out. The PEP process will help us figure out the best way to do that. I definitely intend to implement this in a few different pieces, though - the order of the 4 different fixes in the PEP also summarises what I consider to be a reasonable development strategy as well. Sounds good. I haven't formed a good opinion on the last 2 items yet which was why I didn't comment on them. This morning I did have some interesting thoughts.* *It seems I'm starting to wake up with python code in my head. (Is that good?) In some cases, it's outside of the box solutions, but not always good ones. ;-) Anyway... this mornings fuzzy python thoughts were along the lines of... On one hand, Python tries to make it so each objects source/parent info is reachable from the object when possible. The nice thing about that is, it can be used to create a tree of where everything is. That doesn't work *easily* at the module level due to modules being external OS entities that can be reused in different contexts. On the other hand, a modules import order is somewhat more dynamic compared to class inheritance order, so would it be better to have that info in a separate place rather than with each module? Adding a second references to the '__main__' module begins to blur the purpose of sys.modules from being a place to keep imported modules to a place that also has some ordering information. (Practicality over purity?, Or an indication of what direction to go in?) And, if you ask for keys, items, or values, you will need to filter out '__main__' to avoid the double reference. So I was thinking, what if we make sys.modules a little smarter. The negative of that is, it would no longer be a simple dictionary object. First idea ... Add a __main__ attribute to sys.modules to hold the name of the main module. Override modules.__setitem__, so it will catch '__main__' and set modules.__main__ to the name of the module, and put the module in under its real name instead of '__main__'. Override modules.__getitem__, so it will catch '__main__' and return self[self.__main__]. Then keys(), values(), and items(), will not have the doubled main module references in them. The name of the main module is easy to get. ie... sys.modules.__main__ sys.modules[__name__] will still return the correct module if __name__ == "__main__". Second (more ambitious and less thought out) idea ... Extend sys.modules so it is actually is a tree structure. The '__main__' module would be at the top of the tree. Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Sat, Mar 5, 2011 at 3:08 AM, Toshio Kuratomi wrote: > Some of them can be annoying as hell when dealing with a system that also > installs multiple versions of a module. But one could argue that's the > fault of setuptools' version handling rather than the entry-points > handling. Agreed; I don't use setuptools to manage versions of packages. I've found zc.buildout much nicer to work with, and entirely predictable when applied properly. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Fri, Mar 04, 2011 at 12:56:16PM -0500, Fred Drake wrote: > On Fri, Mar 4, 2011 at 12:35 PM, Michael Foord > wrote: > > That (below) is not distutils it is setuptools. distutils just uses > > `scripts=[...]`, which annoyingly *doesn't* work with setuptools. > > Right; distutils scripts are just sad. > > OTOH, entry-point based scripts are something setuptools got very, > very right. Probably not perfect, but... I've not yet needed anything > different in practice. > Some of them can be annoying as hell when dealing with a system that also installs multiple versions of a module. But one could argue that's the fault of setuptools' version handling rather than the entry-points handling. -Toshio pgpUBRcxfWp3n.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Sat, Mar 5, 2011 at 2:40 PM, Ron Adam wrote: >> Fixing direct execution inside packages > > +1 on both of these. > > I don't see any major problems with these. Any minor issues can be worked > out. > > I suggest separating these two items out into their own PEP, and doing them > first. This first draft includes everything I *want* to do. If I have to scale the proposal back to get it accepted because I can't convince enough other people it's a good idea... well, even Guido isn't immune to that part of the PEP process (cf. PEP 340). It wouldn't be the first PEP to have a "Rejected Ideas" section, and I'm sure it wouldn't be the last. I definitely intend to implement this in a few different pieces, though - the order of the 4 different fixes in the PEP also summarises what I consider to be a reasonable development strategy as well. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On 03/04/2011 09:30 AM, Nick Coghlan wrote: Fixing dual imports of the main module -- Two simple changes are proposed to fix this problem: 1. In ``runpy``, modify the implementation of the ``-m`` switch handling to install the specified module in ``sys.modules`` under both its real name and the name ``__main__``. (Currently it is only installed as the latter) 2. When directly executing a module, install it in ``sys.modules`` under ``os.path.splitext(os.path.basename(__file__))[0]`` as well as under ``__main__``. With the main module also stored under its "real" name, imports will pick it up from the ``sys.modules`` cache rather than reimporting it under a new name. > Fixing direct execution inside packages +1 on both of these. I don't see any major problems with these. Any minor issues can be worked out. I suggest separating these two items out into their own PEP, and doing them first. Cheers, Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Sat, Mar 5, 2011 at 2:59 AM, Thomas Wouters wrote:
> This does nothing to fix another common error: *unwittingly* importing the
> main module under its real name -- for example when you intended to import,
> say, a standard library module of the same name. ('socket.py' is a
> surprisingly common name for a script to experiment with socket
> functionality. Likewise, nowadays, 'twitter.py'.) While the proposed change
> would make it less *broken* to import the main module again, does it make it
> any more *sensible*? Is there really a need to support this? A clear warning
> -- or even error -- would seem much more in place. Doing so is not
> particularly hard: keep a mapping of modules by canonical filename along
> with by modulename, and refuse to add the same file twice. (I'm not talking
> about executing a module inside a package, mind, since that can't shadow a
> stdlib module by accident anymore.)
The proposed behaviour for the top level direct execution is actually
mainly for consistency with the proposed behaviour for direct
execution inside a package.
Your "simple" solution wouldn't work though - several parts of
Python's own test suite would fall in a screaming heap if we did that
(we fiddle with modules a lot in order to exercise various fallback
behaviours, and that fiddling includes importing the same module more
than once with various other modules disabled)
To address the inadvertent shadowing in a more permanent, perhaps we
should aim towards someday making importing the current module via the
usual import syntax an ImportError (or at least an end-user visible
warning), with the advice to use "x = sys.modules[__name__]" instead
if you really intended to do it? Having __source_name__ available in
the main module would greatly simplify that approach.
>> Fixing direct execution inside packages
>> ---
>>
>> To fix this problem, it is proposed that an additional filesystem check be
>> performed before proceeding with direct execution of a ``PY_SOURCE`` or
>> ``PY_COMPILED`` file that has been named on the command line.
>
> This should only happen if the file is a valid import target.
True, things like "foo-test.py" can't be imported (Is that what you
meant by "valid import target"?). This would potentially apply to the
proposed top level change as well.
>> This additional check would look for an ``__init__`` file that is a peer
>> to
>> the specified file with a matching extension (either ``.py``, ``.pyc`` or
>> ``.pyo``, depending what was passed on the command line).
>
> I assume you mean for this to match the normal import rules for packages;
> why not just say that?
I was going back and forth as to whether we should accept an __init__
variant, or only one that matched the exact extension of the file
passed on the command line. On further reflection, just using normal
import semantics makes sense, so I'll revert to that.
> Also, should this consider situations other than the
> vanilla run/import-from-filesystem? Should meta-importers and such get a
> crack at solving this?
No, direct execution is all about vanilla import from the filesystem.
If people want to invoke those other mechanisms, that's what the "-m"
switch and zipfile and directory execution are for (since the latter
two are technically handled as "valid sys.path entry execution", so
the whole path_hooks machinery gets to have a look at them)
>> Fixing pickling without breaking introspection
>> --
>>
>> To fix this problem, it is proposed to add two optional module level
>> attributes: ``__source_name__`` and ``__pickle_name__``.
>>
>> When setting the ``__module__`` attribute on a function or class, the
>> interpreter will be updated to use ``__source_name__`` if defined, falling
>> back to ``__name__`` otherwise.
>>
>> ``__source_name__`` will automatically be set to the main module's "real"
>> name
>> (as described above under the fix to prevent duplicate imports of the main
>> module) by the interpreter. This will fix both pickling and introspection
>> for
>> the main module.
>>
>> It is also proposed that the pickling mechanism for classes and functions
>> be
>> updated to use an optional ``__pickle_module__`` attribute when deciding
>> how
>> to pickle these objects (falling back to the existing ``__module__``
>> attribute if the optional attribute is not defined). When a class or
>> function
>> is defined, this optional attribute will be defined if ``__pickle_name__``
>> is
>> defined at the module level, and left out otherwise. This will allow
>> pseudo-modules to fix pickling without breaking introspection.
>>
>> Other serialisation schemes could add support for this new attribute
>> relatively easily by replacing ``x.__module__`` with ``getattr(x,
>> "__pickle_module__", x.__module__)``.
>>
>> ``pydoc`` and ``inspect`` would also be updated to make appropriate use of
>> the new attributes for any cases not already covered by the abo
Re: [Python-Dev] PEP 395: Module Aliasing
On Sat, Mar 5, 2011 at 3:04 AM, Steven D'Aprano wrote: > I think you mean that sys.path[0] will be set to the directory path. Indeed I did. > Should the current working directory continue to be included in the path > when running a sub-package module? No, it would be similar to the current difference between "python foo.py" and "python -m foo", with the former forcing path[0] to a specific directory, while the latter follows the vagaries of the current working directory. ~/devel$ cat > foo.py import sys print (repr(sys.path[0])) ~/devel$ python foo.py '/home/ncoghlan/devel' ~/devel$ python -m foo '' I'll elaborate on that point in the next update. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Sat, Mar 5, 2011 at 2:22 AM, Fred Drake wrote: > On Fri, Mar 4, 2011 at 10:59 AM, wrote: >> Something to consider here is how this will interact with Python files which >> are _not_ modules. I'm a little uneasy about having sys.modules["trial"] >> refer to the module defined by /usr/bin/trial. > > I've long held the position that modules and scripts are distinct, and > should never share a source file. All the work that's going into > dealing with this just reinforces my position. While I actually have a lot of respect for that point of view (the traps in PEP 395 can be read as a list of justifications for it!), "python -m timeit", "python -m test", "python -m site" and assorted other instances in the stdlib disagree with it. I don't think a purity argument is going to fly here, because the practice of allowing executable modules is just so damn useful for utility functionality where a developer doesn't want to worry about global namespace collisions on the system path . If we turned every module we have in the stdlib with a little snippet of directly executable functionality at the bottom into an installed script, I assume the distro packagers would either outright ignore us or howl the place down with cries of namespace pollution (and they'd be right to do so - the "-m" switch *originated* in a discussion about making timeit easier to invoke on different parallel Python installations without running into namespace collision problems on the system path). Besides, this particular ship really sailed way back in the early days of Python's history when the "if __name__ == '__main__':" convention was established to allow the same source file to behave differently depending on how it was invoked (either as a module or as a script). All PEP 395 is about is trying to make the way that works line up more closely with people's intuitions. However, as a slight tangent, note that one of the positive side effects of the __main__.py file approach to directory and package execution is that it *does* promote keeping the script behaviour and the module behaviour in separate files. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Fri, Mar 4, 2011 at 12:35 PM, Michael Foord wrote: > That (below) is not distutils it is setuptools. distutils just uses > `scripts=[...]`, which annoyingly *doesn't* work with setuptools. Right; distutils scripts are just sad. OTOH, entry-point based scripts are something setuptools got very, very right. Probably not perfect, but... I've not yet needed anything different in practice. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On 04/03/2011 17:24, Barry Warsaw wrote: On Mar 04, 2011, at 05:35 PM, Michael Foord wrote: That (below) is not distutils it is setuptools. distutils just uses `scripts=[...]`, which annoyingly *doesn't* work with setuptools. Sure, but that'll all be moot when distutils2 is integrated into Python 3.3, right? :) Nope. :-) It's still setuptools and not distutils2 format. Michael -Barry -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Mar 04, 2011, at 05:35 PM, Michael Foord wrote: >That (below) is not distutils it is setuptools. distutils just uses >`scripts=[...]`, which annoyingly *doesn't* work with setuptools. Sure, but that'll all be moot when distutils2 is integrated into Python 3.3, right? :) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On 04/03/2011 17:07, Barry Warsaw wrote:
On Mar 04, 2011, at 11:22 AM, Fred Drake wrote:
On Fri, Mar 4, 2011 at 10:59 AM, wrote:
Something to consider here is how this will interact with Python files which
are _not_ modules. I'm a little uneasy about having sys.modules["trial"]
refer to the module defined by /usr/bin/trial.
I've long held the position that modules and scripts are distinct, and
should never share a source file. All the work that's going into
dealing with this just reinforces my position.
I agree. In fact, so do the distribution tools. I don't even include actual
scripts in my packages any more, I just add the right goo to setup.py and let
distutils DTRT:
That (below) is not distutils it is setuptools. distutils just uses
`scripts=[...]`, which annoyingly *doesn't* work with setuptools.
Michael Foord
setup(
# ...
entry_points= {
'console_scripts' : list(scripts),
},
# ...
)
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
--
http://www.voidspace.org.uk/
May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Mar 04, 2011, at 11:22 AM, Fred Drake wrote:
>On Fri, Mar 4, 2011 at 10:59 AM, wrote:
>> Something to consider here is how this will interact with Python files which
>> are _not_ modules. I'm a little uneasy about having sys.modules["trial"]
>> refer to the module defined by /usr/bin/trial.
>
>I've long held the position that modules and scripts are distinct, and
>should never share a source file. All the work that's going into
>dealing with this just reinforces my position.
I agree. In fact, so do the distribution tools. I don't even include actual
scripts in my packages any more, I just add the right goo to setup.py and let
distutils DTRT:
setup(
# ...
entry_points= {
'console_scripts' : list(scripts),
},
# ...
)
-Barry
signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
Nick Coghlan wrote: Fixing direct execution inside packages --- To fix this problem, it is proposed that an additional filesystem check be performed before proceeding with direct execution of a ``PY_SOURCE`` or ``PY_COMPILED`` file that has been named on the command line. This additional check would look for an ``__init__`` file that is a peer to the specified file with a matching extension (either ``.py``, ``.pyc`` or ``.pyo``, depending what was passed on the command line). If this check fails to find anything, direct execution proceeds as usual. If, however, it finds something, execution is handed over to a helper function in the ``runpy`` module that ``runpy.run_path`` also invokes in the same circumstances. That function will walk back up the directory hierarchy from the supplied path, looking for the first directory that doesn't contain an ``__init__`` file. Once that directory is found, it will be set to ``sys.path[0]``, I think you mean that sys.path[0] will be set to the directory path. Should the current working directory continue to be included in the path when running a sub-package module? -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Fri, Mar 4, 2011 at 07:30, Nick Coghlan wrote:
> Fixing dual imports of the main module
> --
>
> Two simple changes are proposed to fix this problem:
>
> 1. In ``runpy``, modify the implementation of the ``-m`` switch handling to
> install the specified module in ``sys.modules`` under both its real name
> and the name ``__main__``. (Currently it is only installed as the latter)
> 2. When directly executing a module, install it in ``sys.modules`` under
> ``os.path.splitext(os.path.basename(__file__))[0]`` as well as under
> ``__main__``.
>
> With the main module also stored under its "real" name, imports will pick
> it
> up from the ``sys.modules`` cache rather than reimporting it under a new
> name.
>
This does nothing to fix another common error: *unwittingly* importing the
main module under its real name -- for example when you intended to import,
say, a standard library module of the same name. ('socket.py' is a
surprisingly common name for a script to experiment with socket
functionality. Likewise, nowadays, 'twitter.py'.) While the proposed change
would make it less *broken* to import the main module again, does it make it
any more *sensible*? Is there really a need to support this? A clear warning
-- or even error -- would seem much more in place. Doing so is not
particularly hard: keep a mapping of modules by canonical filename along
with by modulename, and refuse to add the same file twice. (I'm not talking
about executing a module inside a package, mind, since that can't shadow a
stdlib module by accident anymore.)
Fixing direct execution inside packages
> ---
>
> To fix this problem, it is proposed that an additional filesystem check be
> performed before proceeding with direct execution of a ``PY_SOURCE`` or
> ``PY_COMPILED`` file that has been named on the command line.
>
This should only happen if the file is a valid import target.
> This additional check would look for an ``__init__`` file that is a peer to
> the specified file with a matching extension (either ``.py``, ``.pyc`` or
> ``.pyo``, depending what was passed on the command line).
>
I assume you mean for this to match the normal import rules for packages;
why not just say that? Also, should this consider situations other than the
vanilla run/import-from-filesystem? Should meta-importers and such get a
crack at solving this?
> If this check fails to find anything, direct execution proceeds as usual.
>
> If, however, it finds something, execution is handed over to a
> helper function in the ``runpy`` module that ``runpy.run_path`` also
> invokes
> in the same circumstances. That function will walk back up the
> directory hierarchy from the supplied path, looking for the first directory
> that doesn't contain an ``__init__`` file. Once that directory is found, it
> will be set to ``sys.path[0]``, ``sys.argv[0]`` will be set to ``-m`` and
> ``runpy._run_module_as_main`` will be invoked with the appropriate module
> name (as calculated based on the original filename and the directories
> traversed while looking for a directory without an ``__init__`` file.
>
>
> Fixing pickling without breaking introspection
> --
>
> To fix this problem, it is proposed to add two optional module level
> attributes: ``__source_name__`` and ``__pickle_name__``.
>
> When setting the ``__module__`` attribute on a function or class, the
> interpreter will be updated to use ``__source_name__`` if defined, falling
> back to ``__name__`` otherwise.
>
> ``__source_name__`` will automatically be set to the main module's "real"
> name
> (as described above under the fix to prevent duplicate imports of the main
> module) by the interpreter. This will fix both pickling and introspection
> for
> the main module.
>
> It is also proposed that the pickling mechanism for classes and functions
> be
> updated to use an optional ``__pickle_module__`` attribute when deciding
> how
> to pickle these objects (falling back to the existing ``__module__``
> attribute if the optional attribute is not defined). When a class or
> function
> is defined, this optional attribute will be defined if ``__pickle_name__``
> is
> defined at the module level, and left out otherwise. This will allow
> pseudo-modules to fix pickling without breaking introspection.
>
> Other serialisation schemes could add support for this new attribute
> relatively easily by replacing ``x.__module__`` with ``getattr(x,
> "__pickle_module__", x.__module__)``.
>
> ``pydoc`` and ``inspect`` would also be updated to make appropriate use of
> the new attributes for any cases not already covered by the above rules for
> setting ``__module__``.
>
Is this cornercase really worth polluting the module namespace with more
confusing __*__ names? It seems more sensible to me to simply make pickle
refuse to operate on classes and functions defined in __main__. It wouldn't
even be the least
Re: [Python-Dev] PEP 395: Module Aliasing
Fred Drake wrote: On Fri, Mar 4, 2011 at 10:59 AM, wrote: Something to consider here is how this will interact with Python files which are _not_ modules. I'm a little uneasy about having sys.modules["trial"] refer to the module defined by /usr/bin/trial. I've long held the position that modules and scripts are distinct, and should never share a source file. All the work that's going into dealing with this just reinforces my position. Perhaps I've misunderstood something, but it seems to me that three out of the five problems described in the PEP have nothing to do with modules and scripts sharing the same file. -- Steven ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On Fri, Mar 4, 2011 at 10:59 AM, wrote: > Something to consider here is how this will interact with Python files which > are _not_ modules. I'm a little uneasy about having sys.modules["trial"] > refer to the module defined by /usr/bin/trial. I've long held the position that modules and scripts are distinct, and should never share a source file. All the work that's going into dealing with this just reinforces my position. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 395: Module Aliasing
On 03:30 pm, ncogh...@gmail.com wrote: Fixing dual imports of the main module -- Two simple changes are proposed to fix this problem: 1. In ``runpy``, modify the implementation of the ``-m`` switch handling to install the specified module in ``sys.modules`` under both its real name and the name ``__main__``. (Currently it is only installed as the latter) 2. When directly executing a module, install it in ``sys.modules`` under ``os.path.splitext(os.path.basename(__file__))[0]`` as well as under ``__main__``. With the main module also stored under its "real" name, imports will pick it up from the ``sys.modules`` cache rather than reimporting it under a new name. Something to consider here is how this will interact with Python files which are _not_ modules. I'm a little uneasy about having sys.modules["trial"] refer to the module defined by /usr/bin/trial. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 395: Module Aliasing
There are a bunch of quirks that relate to main modules and pseudo-packages (like the Python 2.7 and 3.2 style unittest) that really don't need to be as annoying as they are. This PEP proposes a few different tricks that should together allow most of those annoyances to be eliminated. Full text below, or you can read a nicely formatted version in the usual place: http://www.python.org/dev/peps/pep-0395/ Cheers, Nick. PEP: 395 Title: Module Aliasing Version: $Revision: 88748 $ Last-Modified: $Date: 2011-03-05 01:26:35 +1000 (Sat, 05 Mar 2011) $ Author: Nick Coghlan Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 4-Mar-2011 Python-Version: 3.3 Post-History: N/A Abstract This PEP proposes new mechanisms that eliminate some longstanding traps for the unwary when dealing with Python's import system, the pickle module and introspection interfaces. What's in a ``__name__``? = Over time, a module's ``__name__`` attribute has come to be used to handle a number of different tasks. The key use cases identified for this module attribute are: 1. Flagging the main module in a program, using the ``if __name__ == "__main__":`` convention. 2. As the starting point for relative imports 3. To identify the location of function and class definitions within the running application 4. To identify the location of classes for serialisation into pickle objects which may be shared with other interpreter instances Traps for the Unwary The overloading of the semantics of ``__name__`` have resulted in several traps for the unwary. These traps can be quite annoying in practice, as they are highly unobvious and can cause quite confusing behaviour. A lot of the time, you won't even notice them, which just makes them all the more surprising when they do come up. Importing the main module twice --- The most venerable of these traps is the issue of (effectively) importing ``__main__`` twice. This occurs when the main module is also imported under its real name, effectively creating two instances of the same module under different names. This problem used to be significantly worse due to implicit relative imports from the main module, but the switch to allowing only absolute imports and explicit relative imports means this issue is now restricted to affecting the main module itself. Why are my relative imports broken? --- PEP 366 defines a mechanism that allows relative imports to work correctly when a module inside a package is executed via the ``-m`` switch. Unfortunately, many users still attempt to directly execute scripts inside packages. While this no longer silently does the wrong thing by creating duplicate copies of peer modules due to implicit relative imports, it now fails noisily at the first explicit relative import, even though the interpreter actually has sufficient information available on the filesystem to make it work properly. In a bit of a pickle Something many users may not realise is that the ``pickle`` module serialises objects based on the ``__name__`` of the containing module. So objects defined in ``__main__`` are pickled that way, and won't be unpickled correctly by another python instance that only imported that module instead of running it directly. Thus the advice from many Python veterans to do as little as possible in the ``__main__`` module in any application that involves any form of object serialisation and persistence. Similarly, when creating a pseudo-module\*, pickles rely on the name of the module where a class is actually defined, rather than the officially documented location for that class in the module hierarchy. While this PEP focuses specifically on ``pickle`` as the principal serialisation scheme in the standard library, this issue may also affect other mechanisms that support serialisation of arbitrary class instances. \*For the purposes of this PEP, a "pseudo-module" is a package designed like the Python 3.2 ``unittest`` and ``concurrent.futures`` packages. These packages are documented as if they were single modules, but are in fact internally implemented as a package. This is *supposed* to be an implementation detail that users and other implementations don't need to worry about, but, thanks to ``pickle``, the details are exposed and effectively become part of the public API. Where's the source? --- Some sophisticated users of the pseudo-module technique described above recognise the problem with implementation details leaking out via the ``pickle`` module, and choose to address it by altering ``__name__`` to refer to the public location for the module before defining any functions or classes (or else by modifying the ``__module__`` attributes of those objects after they have been defined). This approach is effective at eliminating the leakage of information via pickling, but c
