Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Robert Collins]

On 22 April 2015 at 22:13, Christoph Schmitt  wrote:
> Hello again,
>
> since I haven't got any replies yet I'm trying to make myself a bit more
> precise now. I consider the behaviour described in my original posting a
> bug. I posted to this list because the setuptools docs say "Please use the
> distutils-sig mailing list [3] for questions and discussion about

> Test 3)
> DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project
> Result: all modules can be imported

This is correct AFAICT.

the setuptools namespace_packages thing predates PEP-420, and because
PEP-420 namespaces don't interoperate with .pth file based packages
(expecially when you get into interactions between system non-PEP-420
+ virtualenv PEP-420 packages!) changing this is super hard: you'll
guarantee to break many existing installs.

Perhaps there should be a new keyword, but since nothing is needed to
make things work, it seems like it would be rather redundant.

-Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[M.-A. Lemburg]

On 22.04.2015 12:38, Robert Collins wrote:
> On 22 April 2015 at 22:13, Christoph Schmitt  wrote:
>> Hello again,
>>
>> since I haven't got any replies yet I'm trying to make myself a bit more
>> precise now. I consider the behaviour described in my original posting a
>> bug. I posted to this list because the setuptools docs say "Please use the
>> distutils-sig mailing list [3] for questions and discussion about
> 
>> Test 3)
>> DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project
>> Result: all modules can be imported
> 
> This is correct AFAICT.
> 
> the setuptools namespace_packages thing predates PEP-420, and because
> PEP-420 namespaces don't interoperate with .pth file based packages
> (expecially when you get into interactions between system non-PEP-420
> + virtualenv PEP-420 packages!) changing this is super hard: you'll
> guarantee to break many existing installs.
> 
> Perhaps there should be a new keyword, but since nothing is needed to
> make things work, it seems like it would be rather redundant.

You can make use of the namespace_packages keyword argument to setup()
optional depending on which Python version is running it.

I guess that's the only way forward unless you want to break
the package for previous Python versions.

However, doing so may be hard for namespaces which are used
by a lot of packages.

Perhaps setuptools could simply ignore the keyword for
Python 3.3+ and then rely on PEP 420 to get things working
in more or less the same way:

https://www.python.org/dev/peps/pep-0420/

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 22 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Wichert Akkerman]

On Wed, Apr 22, 2015 at 12:38 PM, Robert Collins 
wrote:

> On 22 April 2015 at 22:13, Christoph Schmitt 
> wrote:
> > Hello again,
> >
> > since I haven't got any replies yet I'm trying to make myself a bit more
> > precise now. I consider the behaviour described in my original posting a
> > bug. I posted to this list because the setuptools docs say "Please use
> the
> > distutils-sig mailing list [3] for questions and discussion about
>
> > Test 3)
> > DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project
> > Result: all modules can be imported
>
> This is correct AFAICT.
>
> the setuptools namespace_packages thing predates PEP-420, and because
> PEP-420 namespaces don't interoperate with .pth file based packages
> (expecially when you get into interactions between system non-PEP-420
> + virtualenv PEP-420 packages!) changing this is super hard: you'll
> guarantee to break many existing installs.
>

If I remember things correctly I tried that a while ago, but it resulted in
setuptools generating an sdist without any of the namespace packages.

 Wichert.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[M.-A. Lemburg]

[adding list back on CC]

On 22.04.2015 16:11, Christoph Schmitt wrote:
> Am 2015-04-22 12:59, schrieb M.-A. Lemburg:
>> On 22.04.2015 12:38, Robert Collins wrote:
>>> On 22 April 2015 at 22:13, Christoph Schmitt  wrote:
 Hello again,

 since I haven't got any replies yet I'm trying to make myself a bit more
 precise now. I consider the behaviour described in my original posting a
 bug. I posted to this list because the setuptools docs say "Please use the
 distutils-sig mailing list [3] for questions and discussion about
>>>
 Test 3)
 DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project
 Result: all modules can be imported
>>>
>>> This is correct AFAICT.
>>>
>>> the setuptools namespace_packages thing predates PEP-420, and because
>>> PEP-420 namespaces don't interoperate with .pth file based packages
>>> (expecially when you get into interactions between system non-PEP-420
>>> + virtualenv PEP-420 packages!) changing this is super hard: you'll
>>> guarantee to break many existing installs.
>>>
>>> Perhaps there should be a new keyword, but since nothing is needed to
>>> make things work, it seems like it would be rather redundant.
>>
>> You can make use of the namespace_packages keyword argument to setup()
>> optional depending on which Python version is running it.
>>
>> I guess that's the only way forward unless you want to break
>> the package for previous Python versions.
>>
>> However, doing so may be hard for namespaces which are used
>> by a lot of packages.
>>
>> Perhaps setuptools could simply ignore the keyword for
>> Python 3.3+ and then rely on PEP 420 to get things working
>> in more or less the same way:
>>
>> https://www.python.org/dev/peps/pep-0420/
> 
> I would be fine with declaring namespace_packages conditionally. But doesn't 
> this affect sdist in
> another way than install (or pip install)? If an sdist intended for use with 
> Python < 3.3 is created
> with Python >= 3.3, the included metadata (egg-info) would look different (I 
> don't know if pip
> relies on egg-info or setup.py).

The egg-info generated by setup.py at install time :-)

> That would apply also if namespace_packages would be ignored
> automatically for Python >= 3.3 as you proposed.
> 
> As a consequence, two distributions were neccessary. One with 
> namespace_packages declared and
> containing an __init__.py (with pkg_resources.declare_namespace) and another 
> one without those
> additions. But how does setuptools figure out to leave out the __init__.py 
> for non-declard namespace
> packages in the latter case?

Like I mentioned above: it's probably better for setuptools to handle
this in a consistent way rather than changing all setup.pys.

One detail I'm not sure about is how compatible the two namespace
package techniques really are. The PEP 420 hand waves over possible
differences and only addresses pkgutil, not the setuptools
approach, which is by far more common. For most simply applications
not relying on any of the advanced features, they will most likely
be compatible.

I guess some experiments are necessary to figure that out.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 22 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Robert Collins]

On 23 April 2015 at 03:09, M.-A. Lemburg  wrote:
> [adding list back on CC]
>
> On 22.04.2015 16:11, Christoph Schmitt wrote:
>> Am 2015-04-22 12:59, schrieb M.-A. Lemburg:
>>> On 22.04.2015 12:38, Robert Collins wrote:


>> That would apply also if namespace_packages would be ignored
>> automatically for Python >= 3.3 as you proposed.
>>
>> As a consequence, two distributions were neccessary. One with 
>> namespace_packages declared and
>> containing an __init__.py (with pkg_resources.declare_namespace) and another 
>> one without those
>> additions. But how does setuptools figure out to leave out the __init__.py 
>> for non-declard namespace
>> packages in the latter case?
>
> Like I mentioned above: it's probably better for setuptools to handle
> this in a consistent way rather than changing all setup.pys.

I agree, but consider this situation - on any PEP-420 supporting python

Two packages: name.A and name.B. name.A already installed on the
machine systemwide using old-style namespace path hacks, and then do a
wheel install of name.B.

If the wheel for name.B was built expecting PEP-420, it won't be
importable after install (because the path manipulation that sets up
name as an old-style namespace happens in site.py).

If the wheel was built expecting old-style namespaces, but A was
installed using PEP-420, then A won't be installable after B is
installed (same reason).

The point of splitting the place the two are installed is to show that
the user may not be able to fix the existing install.

So - pip would have to a) detect both styles of package, b)
automatically install all installed-but-wrong-style versions to match
the site installed ones. And if any of the packages in the namespace
only support legacy, everything would be clamped down to legacy.

> One detail I'm not sure about is how compatible the two namespace
> package techniques really are. The PEP 420 hand waves over possible
> differences and only addresses pkgutil, not the setuptools
> approach, which is by far more common. For most simply applications
> not relying on any of the advanced features, they will most likely
> be compatible.

They are entirely incompatible.

> I guess some experiments are necessary to figure that out.

I spent a week last year debugging issues within openstack with
namespace packages, local tree imports of same, and both pure venv and
split system and venv environments.

Some key interesting things:
 - the setuptools pth files aren't fully venv aware today -
potentially fixable but the resulting pth file is fugly, so I decided
not worth it.
 - local tree imports work a heck of a lot better in tox etc with
PEP-420 namespaces
 - PEP-420 namespaces can work on older pythons with importlib, but
 - PEP-420 and legacy packages being mixed for one namespace doesn't
work at all today - in principle fixable via changes to both
setuptools and importlib - but it was about here that the other
openstack folk said 'ok wow, lets just stop using namespace packages
:)

I think its a -lot- easier to reason about these things as two
entirely separate features.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[M.-A. Lemburg]

On 22.04.2015 21:08, Robert Collins wrote:
> On 23 April 2015 at 03:09, M.-A. Lemburg  wrote:
>> [adding list back on CC]
>>
>> On 22.04.2015 16:11, Christoph Schmitt wrote:
>>> Am 2015-04-22 12:59, schrieb M.-A. Lemburg:
 On 22.04.2015 12:38, Robert Collins wrote:
> 
> 
>>> That would apply also if namespace_packages would be ignored
>>> automatically for Python >= 3.3 as you proposed.
>>>
>>> As a consequence, two distributions were neccessary. One with 
>>> namespace_packages declared and
>>> containing an __init__.py (with pkg_resources.declare_namespace) and 
>>> another one without those
>>> additions. But how does setuptools figure out to leave out the __init__.py 
>>> for non-declard namespace
>>> packages in the latter case?
>>
>> Like I mentioned above: it's probably better for setuptools to handle
>> this in a consistent way rather than changing all setup.pys.
> 
> I agree, but consider this situation - on any PEP-420 supporting python
> 
> Two packages: name.A and name.B. name.A already installed on the
> machine systemwide using old-style namespace path hacks, and then do a
> wheel install of name.B.
> 
> If the wheel for name.B was built expecting PEP-420, it won't be
> importable after install (because the path manipulation that sets up
> name as an old-style namespace happens in site.py).
> 
> If the wheel was built expecting old-style namespaces, but A was
> installed using PEP-420, then A won't be installable after B is
> installed (same reason).
> 
> The point of splitting the place the two are installed is to show that
> the user may not be able to fix the existing install.
> 
> So - pip would have to a) detect both styles of package, b)
> automatically install all installed-but-wrong-style versions to match
> the site installed ones. And if any of the packages in the namespace
> only support legacy, everything would be clamped down to legacy.

I don't think support mixed setups is really a practical option.

Either the namespace package is legacy all the way, or it
isn't and uses PEP 420.

Wouldn't it be possible for setuptools or pip to work this out
depending on the Python version ?

Python < 3.3: use legacy system for everything
Python >= 3.3: use PEP 420 for everything (and remove __init__.py
  files at install time as necessary)

>> One detail I'm not sure about is how compatible the two namespace
>> package techniques really are. The PEP 420 hand waves over possible
>> differences and only addresses pkgutil, not the setuptools
>> approach, which is by far more common. For most simply applications
>> not relying on any of the advanced features, they will most likely
>> be compatible.
> 
> They are entirely incompatible.

Oh, I meant from a functionality point of view, not the
technology side.

They both allow installing packages in different directory
trees and that's the only feature most namespace packages
use.

>> I guess some experiments are necessary to figure that out.
> 
> I spent a week last year debugging issues within openstack with
> namespace packages, local tree imports of same, and both pure venv and
> split system and venv environments.
> 
> Some key interesting things:
>  - the setuptools pth files aren't fully venv aware today -
> potentially fixable but the resulting pth file is fugly, so I decided
> not worth it.
>  - local tree imports work a heck of a lot better in tox etc with
> PEP-420 namespaces
>  - PEP-420 namespaces can work on older pythons with importlib, but
>  - PEP-420 and legacy packages being mixed for one namespace doesn't
> work at all today - in principle fixable via changes to both
> setuptools and importlib - but it was about here that the other
> openstack folk said 'ok wow, lets just stop using namespace packages
> :)

It's certainly easier to not use namespace packages and simply
install packages into the same tree. The main reason for
namespace packages in setuptools was the desire to stuff everything
into ZIP files (the eggs) or egg directories, even when installed,
to simplify uninstalls.

As soon as you drop that requirement you can have the package manager
deal with the complexities of having multiple packages share the
same directory in site-packages. That's solvable as pip, RPM,
apt, et al. show :-)

But ok, that doesn't solve the issue of support namespace
packages if the developers want to use them ;-)

> I think its a -lot- easier to reason about these things as two
> entirely separate features.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Apr 22 2015)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> mxODBC Plone/Zope Database Adapter ...   http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pa

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Eric V. Smith]

On 04/22/2015 03:25 PM, M.-A. Lemburg wrote:
> On 22.04.2015 21:08, Robert Collins wrote:
>> So - pip would have to a) detect both styles of package, b)
>> automatically install all installed-but-wrong-style versions to match
>> the site installed ones. And if any of the packages in the namespace
>> only support legacy, everything would be clamped down to legacy.
> 
> I don't think support mixed setups is really a practical option.

Isn't it supposed to be a supported feature?
https://www.python.org/dev/peps/pep-0420/#migrating-from-legacy-namespace-packages

I realize I should know this, but it's been a while since I wrote it.
And I'd swear there are test for this, but I can't find them right now.
But I'm away from home and will have more time to research this later.

Eric.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Robert Collins]

On 23 April 2015 at 08:16, Eric V. Smith  wrote:
> On 04/22/2015 03:25 PM, M.-A. Lemburg wrote:
>> On 22.04.2015 21:08, Robert Collins wrote:
>>> So - pip would have to a) detect both styles of package, b)
>>> automatically install all installed-but-wrong-style versions to match
>>> the site installed ones. And if any of the packages in the namespace
>>> only support legacy, everything would be clamped down to legacy.
>>
>> I don't think support mixed setups is really a practical option.
>
> Isn't it supposed to be a supported feature?
> https://www.python.org/dev/peps/pep-0420/#migrating-from-legacy-namespace-packages
>
> I realize I should know this, but it's been a while since I wrote it.
> And I'd swear there are test for this, but I can't find them right now.
> But I'm away from home and will have more time to research this later.

I'm not sure if that was done. Certainly the caveat there - no dynamic
path support - is somewhat key for the scenarios I was looking at
(e.g. tests in the current tree using 'setup develop' / pip install -e
.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Robert Collins]

On 23 April 2015 at 07:25, M.-A. Lemburg  wrote:
> On 22.04.2015 21:08, Robert Collins wrote:

> I don't think support mixed setups is really a practical option.
>
> Either the namespace package is legacy all the way, or it
> isn't and uses PEP 420.
>
> Wouldn't it be possible for setuptools or pip to work this out
> depending on the Python version ?

Ah, ok so I think this is the crux - I'm arguing that Python version
isn't a big enough check. Because anything installed with a current
version of setuptools, or any wheel built likewise, is going to not
have that per-Python-version check.

And it seems to me that that implies that bringing in a
per-Python-version check in a new release of setuptools or pip is
going to result in mixed mode installs:

install name.A with setuptools-X [legacy]
upgrade setuptools
install name.B with setuptools-Y [does a Python version check]
-> boom

But perhaps sufficient glue can be written to make it all work.

My personal preferred migration strategy is:
 - have a flag day amongst the cooperating packages that make up the namespace
 - release versions that are all in the new layout in a batch to PyPI.

It would be nice if PEP-426 had a conflicts stanza, so you could say
conflicts: [name.A < version_with_new_X] without that implying that
name.A *should* be installed.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Barry Warsaw]

On Apr 23, 2015, at 07:08 AM, Robert Collins wrote:

>PEP-420 and legacy packages being mixed for one namespace doesn't work at all
>today - in principle fixable via changes to both setuptools and importlib -
>but it was about here that the other openstack folk said 'ok wow, lets just
>stop using namespace packages :)

I think this is actually by design, despite what PEP 420 says.  If you have a
portion that contains an __init__.py, that basically overrides any namespace
portions found on sys.path.  It's all or nothing.

Now, were this comes up for me is in bilingual code.  I have some namespace
packages (e.g. flufl.*) which should live in the same parent package even if
the portions live in distinct directories.  Each package has an __init__.py
that stitches things together for Python 2 (using the various common hack
recipes), but of course installing this package in Python 3 means they aren't
namespace packages, and this makes me sad.

I wish there was some kind of exception or marker I could put in the
flufl/__init__.py files that signaled PEP 420-aware Python 3s to treat it as
if the __init__.py doesn't exist.

In the Debian ecosystem, we solve this with package builder help.  The
standard helpers will actually not include the top-level __init__.py file for
the Python 3 binary package version, so they'll be stitched-together
namespace-ish for Python 3 and straight up PEP 420 namespace packages for
Python 3.

Cheers,
-Barry


pgpoJv7iFNd3J.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Chris Barker]

A note from the peanut gallery:

I  like the idea of namepace packages, but every time I've tried to use
them, I've been stymied -- maybe this PEP will solve that, but...

First -  the issues:

- It somehow seems like a lot of work, details to get right, and
more-than-one-way-to-do-it. But maybe that's all pre- PEP 420

- Last time I tried, I couldn't get them to work with "setup.py develop"

But at the core of this -- Why does it have to be so hard? It seems very
simple to me -- what am I missing?

What are namespace packages? To me, they are a package that serves no other
purpose than to provide a single namespace in which to put other packages.
This makes a lot of sense if you have a bunch of related packages where
users may only require one, or a couple, but not all. And you want to be
able to maintain them, and version control them independently.

But it seem to get this, all we need is:

1) A directory with the top-level name

2) It has an (empty)  __init__.py (so it is a python package)

3) It has other directories in it -- each of these are regular old python
packages -- the ONLY difference is that they are installed under that name

That's it. Done. Now all we need is a way to install these things -- well
that's easy, each sub-package installs itself just like it would, maybe
overwriting the top level directory name and the __init__.py, if another
sub-package has already installed it. But that's OK, because the name is by
definition the same, and the __init__ is empty.

This seems SO SIMPLE. No declaring things all over the place, no dynamic
path manipulation, nothing unusual at all, except the ability to install a
module into a dir without clobbering what might already be in that dir.

What am I missing?

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Donald Stufft]

> On Apr 22, 2015, at 5:25 PM, Chris Barker  wrote:
> 
> A note from the peanut gallery:
> 
> I  like the idea of namepace packages, but every time I've tried to use them, 
> I've been stymied -- maybe this PEP will solve that, but...
> 
> First -  the issues:
> 
> - It somehow seems like a lot of work, details to get right, and 
> more-than-one-way-to-do-it. But maybe that's all pre- PEP 420
> 
> - Last time I tried, I couldn't get them to work with "setup.py develop"
> 
> But at the core of this -- Why does it have to be so hard? It seems very 
> simple to me -- what am I missing?
> 
> What are namespace packages? To me, they are a package that serves no other 
> purpose than to provide a single namespace in which to put other packages. 
> This makes a lot of sense if you have a bunch of related packages where users 
> may only require one, or a couple, but not all. And you want to be able to 
> maintain them, and version control them independently.
> 
> But it seem to get this, all we need is:
> 
> 1) A directory with the top-level name
> 
> 2) It has an (empty)  __init__.py (so it is a python package)
> 
> 3) It has other directories in it -- each of these are regular old python 
> packages -- the ONLY difference is that they are installed under that name
> 
> That's it. Done. Now all we need is a way to install these things -- well 
> that's easy, each sub-package installs itself just like it would, maybe 
> overwriting the top level directory name and the __init__.py, if another 
> sub-package has already installed it. But that's OK, because the name is by 
> definition the same, and the __init__ is empty.
> 
> This seems SO SIMPLE. No declaring things all over the place, no dynamic path 
> manipulation, nothing unusual at all, except the ability to install a module 
> into a dir without clobbering what might already be in that dir.
> 
> What am I missing?


Prior to PEP 420 you needed the dynamic path stuff because sometimes your 
namespace package is split across multiple locations on sys.path. Relying on 
the filesystem as you mentioned only works if you install every single 
namespace package into the same directory.

PEP 420 more or less solves all of the problems with namespace packages, other 
than it’s a Python 3 only feature so most people aren’t going to be willing to 
depend on it.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Eric V. Smith]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 04/22/2015 08:31 PM, Donald Stufft wrote:
> 
>> On Apr 22, 2015, at 5:25 PM, Chris Barker > > wrote:
>> 
>> A note from the peanut gallery:
>> 
>> I  like the idea of namepace packages, but every time I've tried
>> to use them, I've been stymied -- maybe this PEP will solve that,
>> but...
>> 
>> First -  the issues:
>> 
>> - It somehow seems like a lot of work, details to get right, and 
>> more-than-one-way-to-do-it. But maybe that's all pre- PEP 420
>> 
>> - Last time I tried, I couldn't get them to work with "setup.py
>> develop"
>> 
>> But at the core of this -- Why does it have to be so hard? It
>> seems very simple to me -- what am I missing?
>> 
>> What are namespace packages? To me, they are a package that
>> serves no other purpose than to provide a single namespace in
>> which to put other packages. This makes a lot of sense if you
>> have a bunch of related packages where users may only require
>> one, or a couple, but not all. And you want to be able to
>> maintain them, and version control them independently.
>> 
>> But it seem to get this, all we need is:
>> 
>> 1) A directory with the top-level name
>> 
>> 2) It has an (empty)  __init__.py (so it is a python package)
>> 
>> 3) It has other directories in it -- each of these are regular
>> old python packages -- the ONLY difference is that they are
>> installed under that name
>> 
>> That's it. Done. Now all we need is a way to install these things
>> -- well that's easy, each sub-package installs itself just like
>> it would, maybe overwriting the top level directory name and the
>> __init__.py, if another sub-package has already installed it. But
>> that's OK, because the name is by definition the same, and the
>> __init__ is empty.
>> 
>> This seems SO SIMPLE. No declaring things all over the place, no 
>> dynamic path manipulation, nothing unusual at all, except the
>> ability to install a module into a dir without clobbering what
>> might already be in that dir.
>> 
>> What am I missing?
> 
> 
> Prior to PEP 420 you needed the dynamic path stuff because
> sometimes your namespace package is split across multiple locations
> on sys.path. Relying on the filesystem as you mentioned only works
> if you install every single namespace package into the same
> directory.
> 
> PEP 420 more or less solves all of the problems with namespace
> packages, other than it’s a Python 3 only feature so most people
> aren’t going to be willing to depend on it.

Right. The problem is that there are 2 ways to install namespace
packages. Say you have a namespace package foo, with 2 "portions"
named bar and baz (see:
https://www.python.org/dev/peps/pep-0420/#terminology). There are 2
ways to install these 2 portions:

1. in 2 different directories on sys.path, say /somewhere/foo-bar and
/somewhere/foo-baz.

2. in a single /somewhere/foo directory on sys.path, with the files
unioned on top of each other, say /somewhere/foo/bar and
/somewhere/foo/baz.

The problem that PEP 420 tries to solve is: what do I do with
foo/__init__.py? Prior to PEP 420, you'd include a copy in both
portions. This foo/__init__.py would contain the magic incantations to
call extend_path(), or whatever similar thing setuptools supports.

This works great for scenario 1 above. You can install the bar portion
or that baz portion, or both, and everything works fine. Note that all
copies of /somewhere/foo/*/__init__.py have the same contents.

The problem is in the second scenario. This is the scenario bdist_rpm
supports. But which RPM owns /somewhere/foo/__init__.py if both
portions are installed? It's the same file in terms of its contents,
and you'd like them to just overlay each other (with either one
winning), but with RPM and other package managers you can't do this,
because you'd have 2 different packages both owning the same file.

So what PEP 420 does is just delete foo/__init__.py. It's never
needed, by any package. Now you can install all namespace packages
with either scenario 1 or 2 above, and everything just works.

Eric.

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.14 (GNU/Linux)

iQEcBAEBAgAGBQJVOE+pAAoJENxauZFcKtNxEf0H/jgUwkdj0q6CsxMC2UPPQg0o
grjhL2FMVfbqhy74aJ0stJhBQ6+fSFR09b6LJ8va3Ql2iJLyXQVX0Kedhts9Hjud
zpfMxpQdxeKE41QjCjeua5hQjTFpqQofxMmcwmjoDOB89Tn+30K1gatPJ4xzjTRc
Lek5UT4yaZTS6mil61vdPUZMSWbxppJQBI0/EUmK9ps4vW3OfVxHMJK0AbvUbRnN
oXRW+NlEhXL2FtMdaoApQQL1STmsH0+RrRY+XlgGbT2G1LbXNviWMLaookUKwLJd
9V4SjS/dgepO9hqL7glSZU7/3THOpF5548gwzmPKuq65bYkx0XzqjUQzaWE6Guo=
=+J5a
-END PGP SIGNATURE-
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Barry Warsaw]

On Apr 22, 2015, at 02:25 PM, Chris Barker wrote:

>That's it. Done. Now all we need is a way to install these things -- well
>that's easy, each sub-package installs itself just like it would, maybe
>overwriting the top level directory name and the __init__.py, if another
>sub-package has already installed it.

This gets at the heart of the motivation for PEP 420, at least for me with my
distro-developer hat on.  Any single file can only be owned by exactly one
distro-level package.

So take the zope.* packages.  Which distro package owns zope/__init__.py when
you have zope.interfaces, zope.component, etc.?  The answer is, none can,
because you have no idea which one will be installed first.

Distros had to go through a lot of hackery to make this work, but now, we can
just delete zope/__init__.py from *all* distro packages (at least for the
Python 3 versions) and it will just work.

Cheers,
-Barry


pgpXwnmS68cZS.pgp
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[Chris Barker]

On Wed, Apr 22, 2015 at 5:31 PM, Donald Stufft  wrote:

> This seems SO SIMPLE.
>
> ...

> What am I missing?
>
> Prior to PEP 420 you needed the dynamic path stuff because sometimes your
> namespace package is split across multiple locations on sys.path.
>

OK -- sure you'd need it then -- still not sure why we need to scatter
namespace packages all of the file system though -- or why we couldn't do
it the quick and dirty and easy way for most cases, anyway...

PEP 420 more or less solves all of the problems with namespace packages,
> other than it’s a Python 3 only feature so most people aren’t going to be
> willing to depend on it.
>

yeah there's always that -- maybe I'll revisit namespace packages when I've
made the full transition to py3...

On Thu, Apr 23, 2015 at 4:38 PM, Barry Warsaw  wrote:

> This gets at the heart of the motivation for PEP 420, at least for me with
> my
> distro-developer hat on.  Any single file can only be owned by exactly one
> distro-level package.
>

I see -- that' a pain. though it seems to me to be a limitation of the
distro-packaging systems, not a python one -- though maybe we need to work
with it anyway...And distro packages need shared directories already --
seems like a lot of work for an empty __init__.py ;-)

Thanks for the clarification.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip

[PJ Eby]

On Wed, Apr 22, 2015 at 5:20 PM, Robert Collins
 wrote:
> Ah, ok so I think this is the crux - I'm arguing that Python version
> isn't a big enough check. Because anything installed with a current
> version of setuptools, or any wheel built likewise, is going to not
> have that per-Python-version check.
>
> And it seems to me that that implies that bringing in a
> per-Python-version check in a new release of setuptools or pip is
> going to result in mixed mode installs:
>
> install name.A with setuptools-X [legacy]
> upgrade setuptools
> install name.B with setuptools-Y [does a Python version check]
> -> boom
>
> But perhaps sufficient glue can be written to make it all work.

When I wrote PEP 402 (the precursor to 420), the idea was that in a
mixed environment you would need to:

1. Change pkg_resources' namespace system to support non-__init__
directories  (and likewise pkgutil.extend_path)
2. Change easy_install's .pth generation magic to not do any magic or
import a PEP 420 emulation module on old systems
3. Change package building tools to stop injecting __init__.py files

This basically solves the mixed installation problem because if you
have an __init__.py that uses existing magic, the empty dirs get
folded in.  You basically have a transitional state with mixed
__init__ and non-__init__ stuff.  If there happens to be an
__init__.py, then as long as it declares the namespace then the local
*runtime* takes care of making the runtime environment work.  The
*installation* tools don't have to manage mixed modes, they should
just blindly install whatever package they have, and over the long
term the packages all end up shipped without __init__.py's, but the
__init__.py approach will continue to work basically forever.

> My personal preferred migration strategy is:
>  - have a flag day amongst the cooperating packages that make up the namespace
>  - release versions that are all in the new layout in a batch to PyPI.
>
> It would be nice if PEP-426 had a conflicts stanza, so you could say
> conflicts: [name.A < version_with_new_X] without that implying that
> name.A *should* be installed.

This is all *really* unnecessary.  Setuptools has essentially *always*
built non-egg, non-exe binary distributions in a PEP 420-compatible
way (i.e., without __init__.py).  And pkg_resources already builds a
namespace path by asking importers if they can import a package at
that location.  So if PEP 420 importers say "yes" when asked to
find_module('somepkg') in the case of an __init__-less subdirectory
named 'somepkg', then pkg_resources *already* supports mixed-mode
installations under PEP 420, and doesn't even need to be updated!

I haven't checked whether that's the case, but if it is, then the only
thing that setuptools neds to change is its .pth generation magic, to
not do the magic if it's on a PEP 420 platform at runtime, and to stop
including __init__.py's for namespace packages.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig