Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On Wed, Apr 22, 2015 at 5:20 PM, Robert Collins
robe...@robertcollins.net wrote:
Ah, ok so I think this is the crux - I'm arguing that Python version
isn't a big enough check. Because anything installed with a current
version of setuptools, or any wheel built likewise, is going to not
have that per-Python-version check.
And it seems to me that that implies that bringing in a
per-Python-version check in a new release of setuptools or pip is
going to result in mixed mode installs:
install name.A with setuptools-X [legacy]
upgrade setuptools
install name.B with setuptools-Y [does a Python version check]
- boom
But perhaps sufficient glue can be written to make it all work.
When I wrote PEP 402 (the precursor to 420), the idea was that in a
mixed environment you would need to:
1. Change pkg_resources' namespace system to support non-__init__
directories (and likewise pkgutil.extend_path)
2. Change easy_install's .pth generation magic to not do any magic or
import a PEP 420 emulation module on old systems
3. Change package building tools to stop injecting __init__.py files
This basically solves the mixed installation problem because if you
have an __init__.py that uses existing magic, the empty dirs get
folded in. You basically have a transitional state with mixed
__init__ and non-__init__ stuff. If there happens to be an
__init__.py, then as long as it declares the namespace then the local
*runtime* takes care of making the runtime environment work. The
*installation* tools don't have to manage mixed modes, they should
just blindly install whatever package they have, and over the long
term the packages all end up shipped without __init__.py's, but the
__init__.py approach will continue to work basically forever.
My personal preferred migration strategy is:
- have a flag day amongst the cooperating packages that make up the namespace
- release versions that are all in the new layout in a batch to PyPI.
It would be nice if PEP-426 had a conflicts stanza, so you could say
conflicts: [name.A version_with_new_X] without that implying that
name.A *should* be installed.
This is all *really* unnecessary. Setuptools has essentially *always*
built non-egg, non-exe binary distributions in a PEP 420-compatible
way (i.e., without __init__.py). And pkg_resources already builds a
namespace path by asking importers if they can import a package at
that location. So if PEP 420 importers say yes when asked to
find_module('somepkg') in the case of an __init__-less subdirectory
named 'somepkg', then pkg_resources *already* supports mixed-mode
installations under PEP 420, and doesn't even need to be updated!
I haven't checked whether that's the case, but if it is, then the only
thing that setuptools neds to change is its .pth generation magic, to
not do the magic if it's on a PEP 420 platform at runtime, and to stop
including __init__.py's for namespace packages.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On Wed, Apr 22, 2015 at 5:31 PM, Donald Stufft don...@stufft.io wrote: This seems SO SIMPLE. ... What am I missing? Prior to PEP 420 you needed the dynamic path stuff because sometimes your namespace package is split across multiple locations on sys.path. OK -- sure you'd need it then -- still not sure why we need to scatter namespace packages all of the file system though -- or why we couldn't do it the quick and dirty and easy way for most cases, anyway... PEP 420 more or less solves all of the problems with namespace packages, other than it’s a Python 3 only feature so most people aren’t going to be willing to depend on it. yeah there's always that -- maybe I'll revisit namespace packages when I've made the full transition to py3... On Thu, Apr 23, 2015 at 4:38 PM, Barry Warsaw ba...@python.org wrote: This gets at the heart of the motivation for PEP 420, at least for me with my distro-developer hat on. Any single file can only be owned by exactly one distro-level package. I see -- that' a pain. though it seems to me to be a limitation of the distro-packaging systems, not a python one -- though maybe we need to work with it anyway...And distro packages need shared directories already -- seems like a lot of work for an empty __init__.py ;-) Thanks for the clarification. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
[adding list back on CC] On 22.04.2015 16:11, Christoph Schmitt wrote: Am 2015-04-22 12:59, schrieb M.-A. Lemburg: On 22.04.2015 12:38, Robert Collins wrote: On 22 April 2015 at 22:13, Christoph Schmitt dev-maili...@gongy.de wrote: Hello again, since I haven't got any replies yet I'm trying to make myself a bit more precise now. I consider the behaviour described in my original posting a bug. I posted to this list because the setuptools docs say Please use the distutils-sig mailing list [3] for questions and discussion about Test 3) DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project Result: all modules can be imported This is correct AFAICT. the setuptools namespace_packages thing predates PEP-420, and because PEP-420 namespaces don't interoperate with .pth file based packages (expecially when you get into interactions between system non-PEP-420 + virtualenv PEP-420 packages!) changing this is super hard: you'll guarantee to break many existing installs. Perhaps there should be a new keyword, but since nothing is needed to make things work, it seems like it would be rather redundant. You can make use of the namespace_packages keyword argument to setup() optional depending on which Python version is running it. I guess that's the only way forward unless you want to break the package for previous Python versions. However, doing so may be hard for namespaces which are used by a lot of packages. Perhaps setuptools could simply ignore the keyword for Python 3.3+ and then rely on PEP 420 to get things working in more or less the same way: https://www.python.org/dev/peps/pep-0420/ I would be fine with declaring namespace_packages conditionally. But doesn't this affect sdist in another way than install (or pip install)? If an sdist intended for use with Python 3.3 is created with Python = 3.3, the included metadata (egg-info) would look different (I don't know if pip relies on egg-info or setup.py). The egg-info generated by setup.py at install time :-) That would apply also if namespace_packages would be ignored automatically for Python = 3.3 as you proposed. As a consequence, two distributions were neccessary. One with namespace_packages declared and containing an __init__.py (with pkg_resources.declare_namespace) and another one without those additions. But how does setuptools figure out to leave out the __init__.py for non-declard namespace packages in the latter case? Like I mentioned above: it's probably better for setuptools to handle this in a consistent way rather than changing all setup.pys. One detail I'm not sure about is how compatible the two namespace package techniques really are. The PEP 420 hand waves over possible differences and only addresses pkgutil, not the setuptools approach, which is by far more common. For most simply applications not relying on any of the advanced features, they will most likely be compatible. I guess some experiments are necessary to figure that out. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 22 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On 23 April 2015 at 03:09, M.-A. Lemburg m...@egenix.com wrote: [adding list back on CC] On 22.04.2015 16:11, Christoph Schmitt wrote: Am 2015-04-22 12:59, schrieb M.-A. Lemburg: On 22.04.2015 12:38, Robert Collins wrote: That would apply also if namespace_packages would be ignored automatically for Python = 3.3 as you proposed. As a consequence, two distributions were neccessary. One with namespace_packages declared and containing an __init__.py (with pkg_resources.declare_namespace) and another one without those additions. But how does setuptools figure out to leave out the __init__.py for non-declard namespace packages in the latter case? Like I mentioned above: it's probably better for setuptools to handle this in a consistent way rather than changing all setup.pys. I agree, but consider this situation - on any PEP-420 supporting python Two packages: name.A and name.B. name.A already installed on the machine systemwide using old-style namespace path hacks, and then do a wheel install of name.B. If the wheel for name.B was built expecting PEP-420, it won't be importable after install (because the path manipulation that sets up name as an old-style namespace happens in site.py). If the wheel was built expecting old-style namespaces, but A was installed using PEP-420, then A won't be installable after B is installed (same reason). The point of splitting the place the two are installed is to show that the user may not be able to fix the existing install. So - pip would have to a) detect both styles of package, b) automatically install all installed-but-wrong-style versions to match the site installed ones. And if any of the packages in the namespace only support legacy, everything would be clamped down to legacy. One detail I'm not sure about is how compatible the two namespace package techniques really are. The PEP 420 hand waves over possible differences and only addresses pkgutil, not the setuptools approach, which is by far more common. For most simply applications not relying on any of the advanced features, they will most likely be compatible. They are entirely incompatible. I guess some experiments are necessary to figure that out. I spent a week last year debugging issues within openstack with namespace packages, local tree imports of same, and both pure venv and split system and venv environments. Some key interesting things: - the setuptools pth files aren't fully venv aware today - potentially fixable but the resulting pth file is fugly, so I decided not worth it. - local tree imports work a heck of a lot better in tox etc with PEP-420 namespaces - PEP-420 namespaces can work on older pythons with importlib, but - PEP-420 and legacy packages being mixed for one namespace doesn't work at all today - in principle fixable via changes to both setuptools and importlib - but it was about here that the other openstack folk said 'ok wow, lets just stop using namespace packages :) I think its a -lot- easier to reason about these things as two entirely separate features. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On 22.04.2015 21:08, Robert Collins wrote: On 23 April 2015 at 03:09, M.-A. Lemburg m...@egenix.com wrote: [adding list back on CC] On 22.04.2015 16:11, Christoph Schmitt wrote: Am 2015-04-22 12:59, schrieb M.-A. Lemburg: On 22.04.2015 12:38, Robert Collins wrote: That would apply also if namespace_packages would be ignored automatically for Python = 3.3 as you proposed. As a consequence, two distributions were neccessary. One with namespace_packages declared and containing an __init__.py (with pkg_resources.declare_namespace) and another one without those additions. But how does setuptools figure out to leave out the __init__.py for non-declard namespace packages in the latter case? Like I mentioned above: it's probably better for setuptools to handle this in a consistent way rather than changing all setup.pys. I agree, but consider this situation - on any PEP-420 supporting python Two packages: name.A and name.B. name.A already installed on the machine systemwide using old-style namespace path hacks, and then do a wheel install of name.B. If the wheel for name.B was built expecting PEP-420, it won't be importable after install (because the path manipulation that sets up name as an old-style namespace happens in site.py). If the wheel was built expecting old-style namespaces, but A was installed using PEP-420, then A won't be installable after B is installed (same reason). The point of splitting the place the two are installed is to show that the user may not be able to fix the existing install. So - pip would have to a) detect both styles of package, b) automatically install all installed-but-wrong-style versions to match the site installed ones. And if any of the packages in the namespace only support legacy, everything would be clamped down to legacy. I don't think support mixed setups is really a practical option. Either the namespace package is legacy all the way, or it isn't and uses PEP 420. Wouldn't it be possible for setuptools or pip to work this out depending on the Python version ? Python 3.3: use legacy system for everything Python = 3.3: use PEP 420 for everything (and remove __init__.py files at install time as necessary) One detail I'm not sure about is how compatible the two namespace package techniques really are. The PEP 420 hand waves over possible differences and only addresses pkgutil, not the setuptools approach, which is by far more common. For most simply applications not relying on any of the advanced features, they will most likely be compatible. They are entirely incompatible. Oh, I meant from a functionality point of view, not the technology side. They both allow installing packages in different directory trees and that's the only feature most namespace packages use. I guess some experiments are necessary to figure that out. I spent a week last year debugging issues within openstack with namespace packages, local tree imports of same, and both pure venv and split system and venv environments. Some key interesting things: - the setuptools pth files aren't fully venv aware today - potentially fixable but the resulting pth file is fugly, so I decided not worth it. - local tree imports work a heck of a lot better in tox etc with PEP-420 namespaces - PEP-420 namespaces can work on older pythons with importlib, but - PEP-420 and legacy packages being mixed for one namespace doesn't work at all today - in principle fixable via changes to both setuptools and importlib - but it was about here that the other openstack folk said 'ok wow, lets just stop using namespace packages :) It's certainly easier to not use namespace packages and simply install packages into the same tree. The main reason for namespace packages in setuptools was the desire to stuff everything into ZIP files (the eggs) or egg directories, even when installed, to simplify uninstalls. As soon as you drop that requirement you can have the package manager deal with the complexities of having multiple packages share the same directory in site-packages. That's solvable as pip, RPM, apt, et al. show :-) But ok, that doesn't solve the issue of support namespace packages if the developers want to use them ;-) I think its a -lot- easier to reason about these things as two entirely separate features. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 22 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
A note from the peanut gallery: I like the idea of namepace packages, but every time I've tried to use them, I've been stymied -- maybe this PEP will solve that, but... First - the issues: - It somehow seems like a lot of work, details to get right, and more-than-one-way-to-do-it. But maybe that's all pre- PEP 420 - Last time I tried, I couldn't get them to work with setup.py develop But at the core of this -- Why does it have to be so hard? It seems very simple to me -- what am I missing? What are namespace packages? To me, they are a package that serves no other purpose than to provide a single namespace in which to put other packages. This makes a lot of sense if you have a bunch of related packages where users may only require one, or a couple, but not all. And you want to be able to maintain them, and version control them independently. But it seem to get this, all we need is: 1) A directory with the top-level name 2) It has an (empty) __init__.py (so it is a python package) 3) It has other directories in it -- each of these are regular old python packages -- the ONLY difference is that they are installed under that name That's it. Done. Now all we need is a way to install these things -- well that's easy, each sub-package installs itself just like it would, maybe overwriting the top level directory name and the __init__.py, if another sub-package has already installed it. But that's OK, because the name is by definition the same, and the __init__ is empty. This seems SO SIMPLE. No declaring things all over the place, no dynamic path manipulation, nothing unusual at all, except the ability to install a module into a dir without clobbering what might already be in that dir. What am I missing? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On Apr 22, 2015, at 5:25 PM, Chris Barker chris.bar...@noaa.gov wrote: A note from the peanut gallery: I like the idea of namepace packages, but every time I've tried to use them, I've been stymied -- maybe this PEP will solve that, but... First - the issues: - It somehow seems like a lot of work, details to get right, and more-than-one-way-to-do-it. But maybe that's all pre- PEP 420 - Last time I tried, I couldn't get them to work with setup.py develop But at the core of this -- Why does it have to be so hard? It seems very simple to me -- what am I missing? What are namespace packages? To me, they are a package that serves no other purpose than to provide a single namespace in which to put other packages. This makes a lot of sense if you have a bunch of related packages where users may only require one, or a couple, but not all. And you want to be able to maintain them, and version control them independently. But it seem to get this, all we need is: 1) A directory with the top-level name 2) It has an (empty) __init__.py (so it is a python package) 3) It has other directories in it -- each of these are regular old python packages -- the ONLY difference is that they are installed under that name That's it. Done. Now all we need is a way to install these things -- well that's easy, each sub-package installs itself just like it would, maybe overwriting the top level directory name and the __init__.py, if another sub-package has already installed it. But that's OK, because the name is by definition the same, and the __init__ is empty. This seems SO SIMPLE. No declaring things all over the place, no dynamic path manipulation, nothing unusual at all, except the ability to install a module into a dir without clobbering what might already be in that dir. What am I missing? Prior to PEP 420 you needed the dynamic path stuff because sometimes your namespace package is split across multiple locations on sys.path. Relying on the filesystem as you mentioned only works if you install every single namespace package into the same directory. PEP 420 more or less solves all of the problems with namespace packages, other than it’s a Python 3 only feature so most people aren’t going to be willing to depend on it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/22/2015 08:31 PM, Donald Stufft wrote: On Apr 22, 2015, at 5:25 PM, Chris Barker chris.bar...@noaa.gov mailto:chris.bar...@noaa.gov wrote: A note from the peanut gallery: I like the idea of namepace packages, but every time I've tried to use them, I've been stymied -- maybe this PEP will solve that, but... First - the issues: - It somehow seems like a lot of work, details to get right, and more-than-one-way-to-do-it. But maybe that's all pre- PEP 420 - Last time I tried, I couldn't get them to work with setup.py develop But at the core of this -- Why does it have to be so hard? It seems very simple to me -- what am I missing? What are namespace packages? To me, they are a package that serves no other purpose than to provide a single namespace in which to put other packages. This makes a lot of sense if you have a bunch of related packages where users may only require one, or a couple, but not all. And you want to be able to maintain them, and version control them independently. But it seem to get this, all we need is: 1) A directory with the top-level name 2) It has an (empty) __init__.py (so it is a python package) 3) It has other directories in it -- each of these are regular old python packages -- the ONLY difference is that they are installed under that name That's it. Done. Now all we need is a way to install these things -- well that's easy, each sub-package installs itself just like it would, maybe overwriting the top level directory name and the __init__.py, if another sub-package has already installed it. But that's OK, because the name is by definition the same, and the __init__ is empty. This seems SO SIMPLE. No declaring things all over the place, no dynamic path manipulation, nothing unusual at all, except the ability to install a module into a dir without clobbering what might already be in that dir. What am I missing? Prior to PEP 420 you needed the dynamic path stuff because sometimes your namespace package is split across multiple locations on sys.path. Relying on the filesystem as you mentioned only works if you install every single namespace package into the same directory. PEP 420 more or less solves all of the problems with namespace packages, other than it’s a Python 3 only feature so most people aren’t going to be willing to depend on it. Right. The problem is that there are 2 ways to install namespace packages. Say you have a namespace package foo, with 2 portions named bar and baz (see: https://www.python.org/dev/peps/pep-0420/#terminology). There are 2 ways to install these 2 portions: 1. in 2 different directories on sys.path, say /somewhere/foo-bar and /somewhere/foo-baz. 2. in a single /somewhere/foo directory on sys.path, with the files unioned on top of each other, say /somewhere/foo/bar and /somewhere/foo/baz. The problem that PEP 420 tries to solve is: what do I do with foo/__init__.py? Prior to PEP 420, you'd include a copy in both portions. This foo/__init__.py would contain the magic incantations to call extend_path(), or whatever similar thing setuptools supports. This works great for scenario 1 above. You can install the bar portion or that baz portion, or both, and everything works fine. Note that all copies of /somewhere/foo/*/__init__.py have the same contents. The problem is in the second scenario. This is the scenario bdist_rpm supports. But which RPM owns /somewhere/foo/__init__.py if both portions are installed? It's the same file in terms of its contents, and you'd like them to just overlay each other (with either one winning), but with RPM and other package managers you can't do this, because you'd have 2 different packages both owning the same file. So what PEP 420 does is just delete foo/__init__.py. It's never needed, by any package. Now you can install all namespace packages with either scenario 1 or 2 above, and everything just works. Eric. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.14 (GNU/Linux) iQEcBAEBAgAGBQJVOE+pAAoJENxauZFcKtNxEf0H/jgUwkdj0q6CsxMC2UPPQg0o grjhL2FMVfbqhy74aJ0stJhBQ6+fSFR09b6LJ8va3Ql2iJLyXQVX0Kedhts9Hjud zpfMxpQdxeKE41QjCjeua5hQjTFpqQofxMmcwmjoDOB89Tn+30K1gatPJ4xzjTRc Lek5UT4yaZTS6mil61vdPUZMSWbxppJQBI0/EUmK9ps4vW3OfVxHMJK0AbvUbRnN oXRW+NlEhXL2FtMdaoApQQL1STmsH0+RrRY+XlgGbT2G1LbXNviWMLaookUKwLJd 9V4SjS/dgepO9hqL7glSZU7/3THOpF5548gwzmPKuq65bYkx0XzqjUQzaWE6Guo= =+J5a -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On 23 April 2015 at 08:16, Eric V. Smith e...@trueblade.com wrote: On 04/22/2015 03:25 PM, M.-A. Lemburg wrote: On 22.04.2015 21:08, Robert Collins wrote: So - pip would have to a) detect both styles of package, b) automatically install all installed-but-wrong-style versions to match the site installed ones. And if any of the packages in the namespace only support legacy, everything would be clamped down to legacy. I don't think support mixed setups is really a practical option. Isn't it supposed to be a supported feature? https://www.python.org/dev/peps/pep-0420/#migrating-from-legacy-namespace-packages I realize I should know this, but it's been a while since I wrote it. And I'd swear there are test for this, but I can't find them right now. But I'm away from home and will have more time to research this later. I'm not sure if that was done. Certainly the caveat there - no dynamic path support - is somewhat key for the scenarios I was looking at (e.g. tests in the current tree using 'setup develop' / pip install -e . -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On 23 April 2015 at 07:25, M.-A. Lemburg m...@egenix.com wrote: On 22.04.2015 21:08, Robert Collins wrote: I don't think support mixed setups is really a practical option. Either the namespace package is legacy all the way, or it isn't and uses PEP 420. Wouldn't it be possible for setuptools or pip to work this out depending on the Python version ? Ah, ok so I think this is the crux - I'm arguing that Python version isn't a big enough check. Because anything installed with a current version of setuptools, or any wheel built likewise, is going to not have that per-Python-version check. And it seems to me that that implies that bringing in a per-Python-version check in a new release of setuptools or pip is going to result in mixed mode installs: install name.A with setuptools-X [legacy] upgrade setuptools install name.B with setuptools-Y [does a Python version check] - boom But perhaps sufficient glue can be written to make it all work. My personal preferred migration strategy is: - have a flag day amongst the cooperating packages that make up the namespace - release versions that are all in the new layout in a batch to PyPI. It would be nice if PEP-426 had a conflicts stanza, so you could say conflicts: [name.A version_with_new_X] without that implying that name.A *should* be installed. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On Apr 23, 2015, at 07:08 AM, Robert Collins wrote: PEP-420 and legacy packages being mixed for one namespace doesn't work at all today - in principle fixable via changes to both setuptools and importlib - but it was about here that the other openstack folk said 'ok wow, lets just stop using namespace packages :) I think this is actually by design, despite what PEP 420 says. If you have a portion that contains an __init__.py, that basically overrides any namespace portions found on sys.path. It's all or nothing. Now, were this comes up for me is in bilingual code. I have some namespace packages (e.g. flufl.*) which should live in the same parent package even if the portions live in distinct directories. Each package has an __init__.py that stitches things together for Python 2 (using the various common hack recipes), but of course installing this package in Python 3 means they aren't namespace packages, and this makes me sad. I wish there was some kind of exception or marker I could put in the flufl/__init__.py files that signaled PEP 420-aware Python 3s to treat it as if the __init__.py doesn't exist. In the Debian ecosystem, we solve this with package builder help. The standard helpers will actually not include the top-level __init__.py file for the Python 3 binary package version, so they'll be stitched-together namespace-ish for Python 3 and straight up PEP 420 namespace packages for Python 3. Cheers, -Barry pgpoJv7iFNd3J.pgp Description: OpenPGP digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On 04/22/2015 03:25 PM, M.-A. Lemburg wrote: On 22.04.2015 21:08, Robert Collins wrote: So - pip would have to a) detect both styles of package, b) automatically install all installed-but-wrong-style versions to match the site installed ones. And if any of the packages in the namespace only support legacy, everything would be clamped down to legacy. I don't think support mixed setups is really a practical option. Isn't it supposed to be a supported feature? https://www.python.org/dev/peps/pep-0420/#migrating-from-legacy-namespace-packages I realize I should know this, but it's been a while since I wrote it. And I'd swear there are test for this, but I can't find them right now. But I'm away from home and will have more time to research this later. Eric. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
Hello again, since I haven't got any replies yet I'm trying to make myself a bit more precise now. I consider the behaviour described in my original posting a bug. I posted to this list because the setuptools docs say Please use the distutils-sig mailing list [3] for questions and discussion about setuptools, and the setuptools bug tracker [4] ONLY for issues you have confirmed via the list. I have two questions, which I hope some expert here can answer: 1) How do I properly handle PEP420 namespace packages with setuptools and pip (Python = 3.3) assuming the scenario described below? 2) Is the behaviour of setuptools/pip as I encountered it intended (including the solution I found) or is it a bug that should be filed? So here is, with some more detail, what I am trying to do: I am using Python 3.4. There are two projects with modules in the namaspace coolpkg. These implement the sub-packages coolpkg.bar and and coolpkg.foo respectively. Both projects have (as allowed by PEP420) no coolpkg/__init__.py. Both projects have a setup.py using setuptools (15.0) to create a source distribution and will be installed using pip (6.1.1). In addition to that, there is another submodule coolpkg.baz, which will not be packaged/installed using setuptools/pip. Instead the folder containing it, will we added to the PYTHONPATH. Here is the complete layout of the projects and the additional module. project-bar/ project-bar/setup.py project-bar/src project-bar/src/coolpkg project-bar/src/coolpkg/bar project-bar/src/coolpkg/bar/__init__.py project-bar/src/coolpkg/bar/barmod.py project-foo/ project-foo/setup.py project-foo/src project-foo/src/coolpkg project-foo/src/coolpkg/foo project-foo/src/coolpkg/foo/foomod.py project-foo/src/coolpkg/foo/__init__.py shady-folder/ shady-folder/coolpkg shady-folder/coolpkg/baz shady-folder/coolpkg/baz/__init__.py shady-folder/coolpkg/baz/bazmod.py My goal is to have a runtime configuration such that modules coolpkg.foo.foomod, coolpkg.bar.barmod, coolpkg.baz.bazmod can all be imported. Test 1) (just a basic test to verify general setup) Add project-bar/src, project-foo/src and shady-folder manually to sys.path Result: works (obviously) For further tests: Create source distributions with setup.py sdist for project-bar and project-foo, install them with pip and put shady-folder on PYTHONPATH. Declare packages=['coolpkg', 'coopkg.foo'] and packages=['coolpkg', 'coopkg.bar'] in respectice setup.py, since find_packages does not play well with PEP420. Test 2) Delcare namespace_packages=['coolpkg'] in setup.py of each project Result: coolpkg.foo.foomod and coolpkg.bar.barmod can be imported, but importing coolpkg.baz.bazmod fails Suspected reason of failure: pip creates *-nspkg.path files which prevent coolpkg.baz from being found (breaking PEP420) Test 3) DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project Result: all modules can be imported To put it bluntly: The setup of test 2) is the one I would expect to work when writing a setup.py as described by the documentation, but pip unneccessarily creates *-nspkg.path files which undo the whole point of PEP420. The setup of test 3) seems like an ulgy hack as we fool pip into not creating *-nspkg.path files by hiding the existance of the namepsace package from setuptools. To summarize my questions stated above: Which is a bug, which is intended and is there a way to handle PEP420-compliant packages properly with the current versions of setuptools and pip. Kind regards, Christoph Schmitt Attachment: Example with scripts, modiy setup.py files according to test 2) and 3) Am 17.04.2015 um 09:31 schrieb Christoph Schmitt: I am using the newest versions of setuptools (15.0) and pip (6.1.1) with Python 3.4. I wanted to satisfy the following two requirements at the same time, but had some trouble: A) creating and installing two source distribution tarballs which have the same top level namespace package (no __init__.py for namespace package) with setuptools and pip B) having other packages within the same namespace package outside of python/site-packages in the PYTHONPATH (not managed by pip/setuptools) Since Python = 3.3 supports namespace packages out of the box (PEP420) this seemed to me like a straightforward task. However, it turned out to be not. Either, setuptools would not find my modules, pip would not install them properly or requirement B) was broken. The solution that worked for me, was to omit the namespace_packages=['whatever_namespace'] declaration in setup.py (to prevent the creation of *-nspkg.pth files by pip) and not to use find_packages, which does not comply with PEP420 (no packages found). This solution is somewhat counter-intuitive and I am not sure, whether it is an intended/valid configuration of a setup.py file. Definitely, it is not clear from the documentation (which has a section about namespace packages btw.). Since I read
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On 22 April 2015 at 22:13, Christoph Schmitt dev-maili...@gongy.de wrote: Hello again, since I haven't got any replies yet I'm trying to make myself a bit more precise now. I consider the behaviour described in my original posting a bug. I posted to this list because the setuptools docs say Please use the distutils-sig mailing list [3] for questions and discussion about Test 3) DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project Result: all modules can be imported This is correct AFAICT. the setuptools namespace_packages thing predates PEP-420, and because PEP-420 namespaces don't interoperate with .pth file based packages (expecially when you get into interactions between system non-PEP-420 + virtualenv PEP-420 packages!) changing this is super hard: you'll guarantee to break many existing installs. Perhaps there should be a new keyword, but since nothing is needed to make things work, it seems like it would be rather redundant. -Rob ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On 22.04.2015 12:38, Robert Collins wrote: On 22 April 2015 at 22:13, Christoph Schmitt dev-maili...@gongy.de wrote: Hello again, since I haven't got any replies yet I'm trying to make myself a bit more precise now. I consider the behaviour described in my original posting a bug. I posted to this list because the setuptools docs say Please use the distutils-sig mailing list [3] for questions and discussion about Test 3) DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project Result: all modules can be imported This is correct AFAICT. the setuptools namespace_packages thing predates PEP-420, and because PEP-420 namespaces don't interoperate with .pth file based packages (expecially when you get into interactions between system non-PEP-420 + virtualenv PEP-420 packages!) changing this is super hard: you'll guarantee to break many existing installs. Perhaps there should be a new keyword, but since nothing is needed to make things work, it seems like it would be rather redundant. You can make use of the namespace_packages keyword argument to setup() optional depending on which Python version is running it. I guess that's the only way forward unless you want to break the package for previous Python versions. However, doing so may be hard for namespaces which are used by a lot of packages. Perhaps setuptools could simply ignore the keyword for Python 3.3+ and then rely on PEP 420 to get things working in more or less the same way: https://www.python.org/dev/peps/pep-0420/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 22 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On Wed, Apr 22, 2015 at 12:38 PM, Robert Collins robe...@robertcollins.net wrote: On 22 April 2015 at 22:13, Christoph Schmitt dev-maili...@gongy.de wrote: Hello again, since I haven't got any replies yet I'm trying to make myself a bit more precise now. I consider the behaviour described in my original posting a bug. I posted to this list because the setuptools docs say Please use the distutils-sig mailing list [3] for questions and discussion about Test 3) DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project Result: all modules can be imported This is correct AFAICT. the setuptools namespace_packages thing predates PEP-420, and because PEP-420 namespaces don't interoperate with .pth file based packages (expecially when you get into interactions between system non-PEP-420 + virtualenv PEP-420 packages!) changing this is super hard: you'll guarantee to break many existing installs. If I remember things correctly I tried that a while ago, but it resulted in setuptools generating an sdist without any of the namespace packages. Wichert. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
