Re: [Distutils] Bilingual namespace package conundrum
On Jan 13, 2015, at 02:39 PM, Robert Collins wrote: Welcome to hell :) Well, purgatory maybe. I patched the lazr packages I cared about. :) So I dug down ridiculously deep into this when investigating a very similar issue in the openstack space. I have an alternative solution to the ones mentioned already - use importlib2 and patch the importer (e.g. in a local site.py) to allow genuine namespaces to work. (Its nearly there - just needs a little more love I think). I like the idea in theory, but I'm not sure how general of a solution this will be. Not everyone can or will want to install the custom importer just to get real namespace packages in Python 2. Cheers, -Barry pgpa5LfbzNW1u.pgp Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Bilingual namespace package conundrum
On 2 January 2015 at 08:19, Barry Warsaw ba...@python.org wrote:
I hope the following makes sense; I've been a little under the weather. ;)
Apologies in advance for the long data-dump, but I wanted to provide as
complete information as possible.
Welcome to hell :)
I have ported Mailman 3 to Python 3.4[*]. While working on various
development tasks, I noticed something rather strange regarding
(pseudo-)namespace packages which MM3 is dependent on. These libraries work
for both Python 2 and 3, but of course namespace packages work completely
differently in those two versions (as of Python 3.3, i.e. PEP 420). I've been
thinking about how such bilingual packages can play better with PEP 420 when
running under a compatible Python 3 version, and still *seem* to work well in
Python 2. More on that later.
It turns out this isn't actually my problem. Here's what's going wrong.
My py3 branch has a fairly straightforward tox.ini. When I run `tox -e py34`
it builds the hidden .tox/py34 virtualenv, installs all the dependencies into
this, runs the test suite, and all is happy. Since I do most of my
development this way, I never noticed any problems.
But then I wanted to work with the code in a slightly different way, so I
created a virtual environment, activated it, and ran `python setup.py develop`
which appeared to work fine. The problem though is that some packages are not
importable by the venv's python. Let's concentrate on three of these
dependencies, lazr.config, lazr.delegates, and lazr.smtptest. All three give
ImportErrors when trying to import them from the interactive prompt of the
venv's python.
I tried something different to see if I could narrow down the problem, so I
created another venv, activated it, and ran `pip install lazr.config
lazr.delegates lazr.smtptest`. Firing up the venv's interactive python, I can
now import all three of them just fine.
So this tells me that `python setup.py develop` is broken, or at the very
least behaves differently that the `pip install` route. Actually, `python
setup.py install` exhibits the same problem.
Just to be sure Debian's Python 3.4 package isn't introducing any weirdnesses,
all of this work was done with an up-to-date build of Python from hg, using
the 3.4 branch, and the virtualenvs were created with `./python -m venv`
Let's look more closely at part of the site-packages directories of the two
virtualenvs. The one that fails looks like this:
drwxrwxr-x [...] lazr.config-2.0.1-py3.4.egg
drwxrwxr-x [...] lazr.delegates-2.0.1-py3.4.egg
drwxrwxr-x [...] lazr.smtptest-2.0.2-py3.4.egg
All three directories exhibit the same pattern:
lazr.config-2.0.1-py3.4.egg
EGG-INFO
namespace_packages.txt (contains one line saying lazr)
lazr
config
__init__.py (contains the pseudo-namespace boilerplate code[+])
__pycache__
[+] that boilerplate code looks like:
-snip snip-
# This is a namespace package, however under = Python 3.3, let it be a true
# namespace package (i.e. this cruft isn't necessary).
import sys
if sys.hexversion 0x30300f0:
try:
import pkg_resources
pkg_resources.declare_namespace(__name__)
except ImportError:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
-snip snip-
but of course, this doesn't really play nicely with PEP 420 because it's the
mere existence of the __init__.py in the namespace package that prevents PEP
420 from getting invoked. As an aside, Debian's packaging tools will actually
*remove* this __init__.py for declared namespace packages in Python 3, so
under Debian-land, things do work properly. The upstream tools have no such
provision, so it's clear why we're getting the ImportErrors. These three
packages do not share a PEP 420-style namespace, and the sys.hexversion guard
prevents the old-style pseudo-namespace hack from working.
This could potentially be fixable in the upstream packages, but again, more on
that later.
Now let's look at the working venvs:
drwxrwxr-x [...] lazr
drwxrwxr-x [...] lazr.config-2.0.1-py3.4.egg-info
-rw-rw-r-- [...] lazr.config-2.0.1-py3.4-nspkg.pth
drwxrwxr-x [...] lazr.delegates-2.0.1-py3.4.egg-info
-rw-rw-r-- [...] lazr.delegates-2.0.1-py3.4-nspkg.pth
drwxrwxr-x [...] lazr.smtptest-2.0.2-py3.4.egg-info
-rw-rw-r-- [...] lazr.smtptest-2.0.2-py3.4-nspkg.pth
This looks quite different, and digging into the 'lazr' directory shows:
lazr
config
delegates
smtptest
No __init__.py in 'lazr', and the sub-package directories contain just the
code for the appropriate library. The egg-info directories look similar, but
now we also have -nspkg.pth files which contain something like the following:
-snip snip-
import sys, types, os;p = os.path.join(sys._getframe(1).f_locals['sitedir'],
*('lazr',));ie = os.path.exists(os.path.join(p,'__init__.py'));m = not
[Distutils] Bilingual namespace package conundrum
I hope the following makes sense; I've been a little under the weather. ;)
Apologies in advance for the long data-dump, but I wanted to provide as
complete information as possible.
I have ported Mailman 3 to Python 3.4[*]. While working on various
development tasks, I noticed something rather strange regarding
(pseudo-)namespace packages which MM3 is dependent on. These libraries work
for both Python 2 and 3, but of course namespace packages work completely
differently in those two versions (as of Python 3.3, i.e. PEP 420). I've been
thinking about how such bilingual packages can play better with PEP 420 when
running under a compatible Python 3 version, and still *seem* to work well in
Python 2. More on that later.
It turns out this isn't actually my problem. Here's what's going wrong.
My py3 branch has a fairly straightforward tox.ini. When I run `tox -e py34`
it builds the hidden .tox/py34 virtualenv, installs all the dependencies into
this, runs the test suite, and all is happy. Since I do most of my
development this way, I never noticed any problems.
But then I wanted to work with the code in a slightly different way, so I
created a virtual environment, activated it, and ran `python setup.py develop`
which appeared to work fine. The problem though is that some packages are not
importable by the venv's python. Let's concentrate on three of these
dependencies, lazr.config, lazr.delegates, and lazr.smtptest. All three give
ImportErrors when trying to import them from the interactive prompt of the
venv's python.
I tried something different to see if I could narrow down the problem, so I
created another venv, activated it, and ran `pip install lazr.config
lazr.delegates lazr.smtptest`. Firing up the venv's interactive python, I can
now import all three of them just fine.
So this tells me that `python setup.py develop` is broken, or at the very
least behaves differently that the `pip install` route. Actually, `python
setup.py install` exhibits the same problem.
Just to be sure Debian's Python 3.4 package isn't introducing any weirdnesses,
all of this work was done with an up-to-date build of Python from hg, using
the 3.4 branch, and the virtualenvs were created with `./python -m venv`
Let's look more closely at part of the site-packages directories of the two
virtualenvs. The one that fails looks like this:
drwxrwxr-x [...] lazr.config-2.0.1-py3.4.egg
drwxrwxr-x [...] lazr.delegates-2.0.1-py3.4.egg
drwxrwxr-x [...] lazr.smtptest-2.0.2-py3.4.egg
All three directories exhibit the same pattern:
lazr.config-2.0.1-py3.4.egg
EGG-INFO
namespace_packages.txt (contains one line saying lazr)
lazr
config
__init__.py (contains the pseudo-namespace boilerplate code[+])
__pycache__
[+] that boilerplate code looks like:
-snip snip-
# This is a namespace package, however under = Python 3.3, let it be a true
# namespace package (i.e. this cruft isn't necessary).
import sys
if sys.hexversion 0x30300f0:
try:
import pkg_resources
pkg_resources.declare_namespace(__name__)
except ImportError:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
-snip snip-
but of course, this doesn't really play nicely with PEP 420 because it's the
mere existence of the __init__.py in the namespace package that prevents PEP
420 from getting invoked. As an aside, Debian's packaging tools will actually
*remove* this __init__.py for declared namespace packages in Python 3, so
under Debian-land, things do work properly. The upstream tools have no such
provision, so it's clear why we're getting the ImportErrors. These three
packages do not share a PEP 420-style namespace, and the sys.hexversion guard
prevents the old-style pseudo-namespace hack from working.
This could potentially be fixable in the upstream packages, but again, more on
that later.
Now let's look at the working venvs:
drwxrwxr-x [...] lazr
drwxrwxr-x [...] lazr.config-2.0.1-py3.4.egg-info
-rw-rw-r-- [...] lazr.config-2.0.1-py3.4-nspkg.pth
drwxrwxr-x [...] lazr.delegates-2.0.1-py3.4.egg-info
-rw-rw-r-- [...] lazr.delegates-2.0.1-py3.4-nspkg.pth
drwxrwxr-x [...] lazr.smtptest-2.0.2-py3.4.egg-info
-rw-rw-r-- [...] lazr.smtptest-2.0.2-py3.4-nspkg.pth
This looks quite different, and digging into the 'lazr' directory shows:
lazr
config
delegates
smtptest
No __init__.py in 'lazr', and the sub-package directories contain just the
code for the appropriate library. The egg-info directories look similar, but
now we also have -nspkg.pth files which contain something like the following:
-snip snip-
import sys, types, os;p = os.path.join(sys._getframe(1).f_locals['sitedir'],
*('lazr',));ie = os.path.exists(os.path.join(p,'__init__.py'));m = not ie and
sys.modules.setdefault('lazr', types.ModuleType('lazr'));mp = (m or []) and
m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
-snip snip-
Re: [Distutils] Bilingual namespace package conundrum
On Jan 1, 2015, at 4:54 PM, Barry Warsaw ba...@python.org wrote: On Jan 01, 2015, at 03:20 PM, Tres Seaver wrote: That sounds right to me. I never really understood the motivation for PEP 420, but if the presence of that file disables it, then it should ensure the old behavior regardless. The motivation is described in the PEP, but briefly, on (many, most, all?) Linux distros any single file can be owned by exactly one system package. So which package would own e.g. zope/__init__.py? Before PEP 420, the answer is well, it's complicated. After PEP 420, the answer is no package, because that file doesn't exist. This is why the Debian tools delete any stray zope/__init__.py files when they are creating any zope.* binary package for Python 3. I think you're right that removing the version guard will make it work, but they won't be PEP 420 packages. I don't think it's possible to make a bilingual package a pseudo-namespace package under Python 2, but a PEP 420 namespace package under Python = 3.3. Cheers, -Barry ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig I’m pretty sure the problem with setup.py develop and setup.py install is because they are installed as eggs more or less and setuptools probably doesn’t support it. pip install whatever installs it unpacked so it’ll rely on built in importing. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Bilingual namespace package conundrum
On Jan 1, 2015, at 9:11 PM, Tres Seaver tsea...@palladion.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/01/2015 04:57 PM, Donald Stufft wrote: I’m pretty sure the problem with setup.py develop and setup.py install is because they are installed as eggs more or less and setuptools probably doesn’t support it. pip install whatever installs it unpacked so it’ll rely on built in importing. I'm puzzled by that notion: 'setup.py develop' and 'setup.py install' have worked for a decade with namespace packages, as long as the __init__ for them does the Right Thing (using either pkg_resources or pkgutil). What has never worked properly is mixing pip installation of namespace packages with easy_install installation of packages in the same namespace: pip's install in one directory without fixups stragedy screws up the __path__ for the namespace packages installed elsewhere. Why are you puzzled by the notion that something designed to work with a one mechanism for a particular feature probably does not work with a newer, different mechanism for a particular feature? My assumption is that setuptools is ensuring that the __init__.py files are being written (or the nspkg or whatever, I forget which files are used in which cases) which is breaking the PEP420 “implicit” namespace packages. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Bilingual namespace package conundrum
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/01/2015 04:57 PM, Donald Stufft wrote: I’m pretty sure the problem with setup.py develop and setup.py install is because they are installed as eggs more or less and setuptools probably doesn’t support it. pip install whatever installs it unpacked so it’ll rely on built in importing. I'm puzzled by that notion: 'setup.py develop' and 'setup.py install' have worked for a decade with namespace packages, as long as the __init__ for them does the Right Thing (using either pkg_resources or pkgutil). What has never worked properly is mixing pip installation of namespace packages with easy_install installation of packages in the same namespace: pip's install in one directory without fixups stragedy screws up the __path__ for the namespace packages installed elsewhere. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlSl/kEACgkQ+gerLs4ltQ4aYQCcDImoKfLr3Xb+tkFSY9x8Oinh QT8AoNDKmxVoqdX3UMnsUg1ktZbqukZg =CPP5 -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Bilingual namespace package conundrum
On Jan 01, 2015, at 09:14 PM, Donald Stufft wrote: Why are you puzzled by the notion that something designed to work with a one mechanism for a particular feature probably does not work with a newer, different mechanism for a particular feature? My assumption is that setuptools is ensuring that the __init__.py files are being written (or the nspkg or whatever, I forget which files are used in which cases) which is breaking the PEP420 “implicit” namespace packages. Note that in neither case are actual PEP 420 namespace packages being used. It would be nice if both pip and setuptools could handle namespace packages, but neither do. In fact I think if they did, there wouldn't be a problem. The problem comes about because setuptools is installing them as separate eggs which are *not* PEP 420 portions. pip succeeds because it happens to install the portions under the same top-level directory, so the namespace's __init__.py just happens to get overwritten and thus works. Cheers, -Barry pgp0ZRJOihdvm.pgp Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Bilingual namespace package conundrum
On Jan 01, 2015, at 03:20 PM, Tres Seaver wrote: That sounds right to me. I never really understood the motivation for PEP 420, but if the presence of that file disables it, then it should ensure the old behavior regardless. The motivation is described in the PEP, but briefly, on (many, most, all?) Linux distros any single file can be owned by exactly one system package. So which package would own e.g. zope/__init__.py? Before PEP 420, the answer is well, it's complicated. After PEP 420, the answer is no package, because that file doesn't exist. This is why the Debian tools delete any stray zope/__init__.py files when they are creating any zope.* binary package for Python 3. I think you're right that removing the version guard will make it work, but they won't be PEP 420 packages. I don't think it's possible to make a bilingual package a pseudo-namespace package under Python 2, but a PEP 420 namespace package under Python = 3.3. Cheers, -Barry pgp5VR5IonTtK.pgp Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Bilingual namespace package conundrum
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/01/2015 02:19 PM, Barry Warsaw wrote: Maybe the sys.hexversion guards should be removed so that it acts the same way in both Python 2 and Python 3. That sounds right to me. I never really understood the motivation for PEP 420, but if the presence of that file disables it, then it should ensure the old behavior regardless. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlSlq/oACgkQ+gerLs4ltQ5hjgCeKMfXT0DPBS//0y/XAP2npJRW KssAniyBjuuhjVpw0ETola6v6hVuwpiP =BLOQ -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
