Re: [Distutils] nspkg.pth files break $PYTHONPATH overrides
On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote: >See also https://github.com/pypa/pip/issues/3 Hah, yeah. I didn't realize --single-version-externally-managed is implied by --root, and in fact our Debian build scripts often (either explicitly or implicitly) define --root, as is the case in lazr.uri. >Basically prior to PEP420 namespace packages were bad and using them results >in pain sooner or later :( I’m not sure if a good solution yet, perhaps we >can backport PEP420 to PyPI and have namespace packages depend on that? Or maybe do a version check before installing this file? Python 3.3 and newer will have PEP 420 support, and this file makes no sense in that case. (Backporting PEP 420 support to Python 3.2 might be useful, though a pain without importlib. There's no sense in backporting to Python 3.1, IMHO.) -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] nspkg.pth files break $PYTHONPATH overrides
See also https://github.com/pypa/pip/issues/3
Basically prior to PEP420 namespace packages were bad and using them
results in pain sooner or later :( I’m not sure if a good solution yet, perhaps
we can backport PEP420 to PyPI and have namespace packages depend
on that?
On Mar 24, 2014, at 5:48 PM, Barry Warsaw wrote:
> Apologies for cross-posting, but this intersects setuptools and the import
> system, and I wanted to be sure it reached the right audience.
>
> A colleague asked me why a seemingly innocent and common use case for
> developing local versions of system installed packages wasn't working, and I
> was quite perplexed. As I dug into the problem, more questions than answers
> came up. I finally (think! I) figured out what is happening, but not so much
> as to why, or what can/should be done about it.
>
> This person had a local checkout of a package's source, where the package was
> also installed into the system Python. He wanted to be able to set
> $PYTHONPATH so that the local package wins when he tries to import it. E.g.:
>
> % PYTHONPATH=`pwd`/src python3
>
> but this didn't work because despite the setting of PYTHONPATH, the system
> version of the package was always found first. The package in question is
> lazr.uri, although other packages with similar layouts will also suffer the
> same problem, which prevents an easy local development of a newer version of
> the package, aside from being a complete head-scratcher.
>
> The lazr.uri package is intended to be a submodule of the lazr namespace
> package. As such, the lazr/__init__.py has the old style way of declaring a
> namespace package:
>
> try:
>import pkg_resources
>pkg_resources.declare_namespace(__name__)
> except ImportError:
>import pkgutil
>__path__ = pkgutil.extend_path(__path__, __name__)
>
> and its setup.py declares a namespace package:
>
> setup(
>name='lazr.uri',
>version=__version__,
>namespace_packages=['lazr'],
>...
>
> One of the things that the Debian "helper" program does when it builds a
> package for the archive is call `$python setup.py install_egg_info`. It's
> this command that breaks $PYTHONPATH overriding.
>
> install_egg_info looks at the lazr.uri.egg-info/namespace_packages.txt file,
> in which it finds the string 'lazr', and it proceeds to write a
> lazr-uri-1.0.3-py3.4-nspkg.pth file. This causes other strange and unexpected
> things to happen:
>
> % python3
> Python 3.4.0 (default, Mar 22 2014, 22:51:25)
> [GCC 4.8.2] on linux
> Type "help", "copyright", "credits" or "license" for more information.
import sys
sys.modules['lazr']
>
sys.modules['lazr'].__path__
> ['/usr/lib/python3/dist-packages/lazr']
>
> It's completely weird that sys.modules would contain a key for 'lazr' when
> that package was never explicitly imported. Even stranger, because a fake
> module object is stuffed into sys.modules via the .pth file, tracing imports
> with -v gives you no clue as to what's happening. And while
> sys.modules['lazr'] has an __path__, it has no other attributes.
>
> I really don't understand what the purpose of the nspkg.pth file is,
> especially for Python 3 namespace packages.
>
> Here's what the nspkg.pth file contains:
>
> import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'],
> *('lazr',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie
> and sys.modules.setdefault('lazr',types.ModuleType('lazr')); mp = (m or [])
> and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
>
> The __path__ value is important here because even though you've never
> explicitly imported 'lazr', when you *do* explicitly import 'lazr.uri', the
> existing lazr module object's __path__ takes over, and thus the system
> lazr.uri package is found even though both lazr/ and lazr/uri/ should have
> been found earlier on sys.path (yes, sys.path looks exactly as expected).
>
> So the presence of the nspkg.pth file breaks $PYTHONPATH overriding. That
> seems bad. ;)
>
> If you delete the nspkg.path file, then things work as expected, but even this
> is a little misleading!
>
> I think the Debian helper is running install_egg_info as a way to determine
> what namespace packages are defined, so that it can actually *remove* the
> parent's __init__.py file and use PEP 420 style namespace packages. In fact,
> in the Debian python3-lazr.uri binary package, you find no system
> lazr/__init__.py file. This is why removing the nspkg.pth file works.
>
> So I thought, why not conditionally define setup(..., namespace_packages) only
> for Python 2? This doesn't work because the Debian helper will see that no
> namespace packages are defined, and thus it will leave the original
> lazr/__init__.py file in place. This then breaks $PYTHONPATH overriding too
> because of __path__ extension of the pre-PEP 420 code only *appends* the local
> development path. IOW, the system import path is the first element of
[Distutils] nspkg.pth files break $PYTHONPATH overrides
Apologies for cross-posting, but this intersects setuptools and the import
system, and I wanted to be sure it reached the right audience.
A colleague asked me why a seemingly innocent and common use case for
developing local versions of system installed packages wasn't working, and I
was quite perplexed. As I dug into the problem, more questions than answers
came up. I finally (think! I) figured out what is happening, but not so much
as to why, or what can/should be done about it.
This person had a local checkout of a package's source, where the package was
also installed into the system Python. He wanted to be able to set
$PYTHONPATH so that the local package wins when he tries to import it. E.g.:
% PYTHONPATH=`pwd`/src python3
but this didn't work because despite the setting of PYTHONPATH, the system
version of the package was always found first. The package in question is
lazr.uri, although other packages with similar layouts will also suffer the
same problem, which prevents an easy local development of a newer version of
the package, aside from being a complete head-scratcher.
The lazr.uri package is intended to be a submodule of the lazr namespace
package. As such, the lazr/__init__.py has the old style way of declaring a
namespace package:
try:
import pkg_resources
pkg_resources.declare_namespace(__name__)
except ImportError:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
and its setup.py declares a namespace package:
setup(
name='lazr.uri',
version=__version__,
namespace_packages=['lazr'],
...
One of the things that the Debian "helper" program does when it builds a
package for the archive is call `$python setup.py install_egg_info`. It's
this command that breaks $PYTHONPATH overriding.
install_egg_info looks at the lazr.uri.egg-info/namespace_packages.txt file,
in which it finds the string 'lazr', and it proceeds to write a
lazr-uri-1.0.3-py3.4-nspkg.pth file. This causes other strange and unexpected
things to happen:
% python3
Python 3.4.0 (default, Mar 22 2014, 22:51:25)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules['lazr']
>>> sys.modules['lazr'].__path__
['/usr/lib/python3/dist-packages/lazr']
It's completely weird that sys.modules would contain a key for 'lazr' when
that package was never explicitly imported. Even stranger, because a fake
module object is stuffed into sys.modules via the .pth file, tracing imports
with -v gives you no clue as to what's happening. And while
sys.modules['lazr'] has an __path__, it has no other attributes.
I really don't understand what the purpose of the nspkg.pth file is,
especially for Python 3 namespace packages.
Here's what the nspkg.pth file contains:
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'],
*('lazr',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and
sys.modules.setdefault('lazr',types.ModuleType('lazr')); mp = (m or []) and
m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
The __path__ value is important here because even though you've never
explicitly imported 'lazr', when you *do* explicitly import 'lazr.uri', the
existing lazr module object's __path__ takes over, and thus the system
lazr.uri package is found even though both lazr/ and lazr/uri/ should have
been found earlier on sys.path (yes, sys.path looks exactly as expected).
So the presence of the nspkg.pth file breaks $PYTHONPATH overriding. That
seems bad. ;)
If you delete the nspkg.path file, then things work as expected, but even this
is a little misleading!
I think the Debian helper is running install_egg_info as a way to determine
what namespace packages are defined, so that it can actually *remove* the
parent's __init__.py file and use PEP 420 style namespace packages. In fact,
in the Debian python3-lazr.uri binary package, you find no system
lazr/__init__.py file. This is why removing the nspkg.pth file works.
So I thought, why not conditionally define setup(..., namespace_packages) only
for Python 2? This doesn't work because the Debian helper will see that no
namespace packages are defined, and thus it will leave the original
lazr/__init__.py file in place. This then breaks $PYTHONPATH overriding too
because of __path__ extension of the pre-PEP 420 code only *appends* the local
development path. IOW, the system import path is the first element of a
2-element list on lazr.__path__. While the local import path is the second
element, in this case too the local import fails.
It seems like what you want for Python 3 (and we're talking >= 3.2 here) is
for there to be neither a nspkg.pth file, nor the lazr/__init__.py file, and
let PEP 420 do it's thing. In fact if you set things up this way, $PYTHONPATH
overriding works exactly as expected.
Because I don't know why install_egg_info is installing the nspkg.pth file, I
don't know which component needs
Re: [Distutils] setuptools tracker
On Mon, Mar 24, 2014 at 8:46 AM, Nick Coghlan wrote: > On 24 March 2014 20:53, "Martin v. Löwis" wrote: > > Both problems would be resolved by setting the tracker to read-only; > > shutting it down is actually not necessary (although it would slightly > > reduce our maintenance efforts). > > That sounds good to me. > > I've also filed https://bitbucket.org/pypa/setuptools/issue/174/ to > decide on a longer term solution. If Jason decides to review/migrate > issues, it may be necessary to turn developer write access back on to > allow issues to be marked as closed once they have been dealt with > appropriately. > Yep, looks like Jason came to the same conclusion(s) independently, but also wants better banners on the old tracker to alert people to the move. I guess we should move any further discussion to that ticket, since Jason's response time is quicker there than here. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
> Step 3 ("what, if anything, should replace the setup.py CLI as the
> build system abstraction?") [...] research/documentation consolidation
> project at this point than it is a design project.
FYI, I've been experimenting with a design approach for a build system in
distil. Currently it only covers what distutils/setuptools already do - i.e.
build_py functionality + build_ext functionality covering libraries,
extensions, setuptools Features, Cython / Pyrex and Fortran / f2py. Not too
shabby, but clearly that's not enough. Instead of compiler classes (the
abstractions used in distutils / setuptools), distil's approach focuses on
a command line with variable placeholders - this allows just about anything
to be plugged in for custom builds. There's still work to be done on it, but
the initial approach looks promising.
Regards,
Vinay Sajip
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker
On 24 March 2014 20:53, "Martin v. Löwis" wrote: > Am 24.03.14 01:33, schrieb PJ Eby: >> I think it would be a good idea to check with Jason and other PyPA >> volunteers to see if there is anyone else who can handle the moves. I'd >> prefer we didn't lose the history, since my comments on those issues >> (and the closed ones, too) often contain key information about use cases >> and design decisions that may not be available elsewhere, even from my >> memory. ;-) But, since I'm no longer in the lead on development, I >> think it would be better for someone closer to the future of things to >> do the prioritizing of what, if anything, to transfer as an issue or >> keep as documentation. > > Yet alternatively, I could set the tracker to read-only, and keep it up > for any foreseeable future. > > The reason I'm bringing this up is two-fold: > 1. some people started using the tracker to distribute spam > 2. some people apparently still think that the system is active, >and continue reporting issues there. > > Both problems would be resolved by setting the tracker to read-only; > shutting it down is actually not necessary (although it would slightly > reduce our maintenance efforts). That sounds good to me. I've also filed https://bitbucket.org/pypa/setuptools/issue/174/ to decide on a longer term solution. If Jason decides to review/migrate issues, it may be necessary to turn developer write access back on to allow issues to be marked as closed once they have been dealt with appropriately. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker
Am 24.03.14 01:33, schrieb PJ Eby: > I think it would be a good idea to check with Jason and other PyPA > volunteers to see if there is anyone else who can handle the moves. I'd > prefer we didn't lose the history, since my comments on those issues > (and the closed ones, too) often contain key information about use cases > and design decisions that may not be available elsewhere, even from my > memory. ;-) But, since I'm no longer in the lead on development, I > think it would be better for someone closer to the future of things to > do the prioritizing of what, if anything, to transfer as an issue or > keep as documentation. Yet alternatively, I could set the tracker to read-only, and keep it up for any foreseeable future. The reason I'm bringing this up is two-fold: 1. some people started using the tracker to distribute spam 2. some people apparently still think that the system is active, and continue reporting issues there. Both problems would be resolved by setting the tracker to read-only; shutting it down is actually not necessary (although it would slightly reduce our maintenance efforts). Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] waf can be used to compile Python extension modules
On 24 Mar 2014 10:01, "Paul Moore" wrote:
>
> On 23 March 2014 23:13, Nick Coghlan wrote:
> > Now you begin to see the scope of the problem. It's definitely
solvable, but
> > means asking a whole pile of "required, recommended or
distutils-specific?"
> > questions about the existing distutils and setuptools build system :)
> >
> > "pip already relies on it" sets the minimum for the "required"
category, but
> > there's more to a full build system abstraction than what pip currently
> > supports.
>
> OK, I see now. So the ultimate build system will include pip changes
> to supply build-time options in an as-yet unspecified manner.
>
> There's certainly no way I can do all of that myself, I don't have
> remotely the level of experience with complex build requirements. But
> I can probably take the first steps, and leave it to people with the
> experience to add to it. No promises on timescales but I'll see what I
> can do.
>
> One thought. do we want to use a setup.py script as the interface,
> with all its historical baggage, or would we be better using a new
> script name as the "official" interface (with pip falling back to
> equivalent setup.py invocations when that script isn't present, for
> backward compatibility)?
Step 1 is "What does pip currently expect setup.py to support?"
Step 2 is "What other existing features of distutils/setuptools do we think
a reasonable replacement for setup.py should support?" (I don't believe
distutils2 reached the point of addressing this, but that should still be
checked)
Step 3 ("what, if anything, should replace the setup.py CLI as the build
system abstraction?") really depends on the outcome of steps 1 & 2 - this
is more a research/documentation consolidation project at this point than
it is a design project.
Cheers,
Nick.
>
> Paul
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
