Re: Python packaging, dependencies, upstream facilities

[Barry Warsaw]

On Sep 21, 2010, at 03:02 PM, Clint Byrum wrote:

>In the java world, they use maven because it handles this for them.
>They create a maven spec file that says "I need libX, libY, and
>libZ (v1.1)". maven, during the build, goes out and finds libX and
>libY's latest versions, then finds the closest match to libZ v1.1.
>These are placed in jars in the classpath for the project, and viola
>it "just works".

Tools like buildout and virtualenv exist to address issues like this.
Buildout's goal in life for example is to be able to version control
repeatable built environments.  It plays quite nicely with Python's own
package management system (PyPI, aka Cheeseshop), but poorly with the
operating system's package system.  This isn't specific to Debian; a Python
developer on Fedora would feel the same pain.

There have been sessions at probably the last 3 or 4 Pycon sprints to try to
address this, but it's difficult.  I do think that the work that Tarek and
distutils-sig is doing will get us closer, and the more Debian developers that
are involved in the process, the better for us.

I guess, if it were easy it would be done by now. ;)

>But, if you can get the order of the library loading path right,
>then this structure solves Robert's original desire:
>
>/usr/share/pyshared/foo1
>/usr/share/pyshared/foo2
>/usr/share/pyshared/foo => foo2
>/usr/share/pyshared/consumer-of-foo1/lib-packages/foo
>=> /usr/share/pyshared/foo1

My sense is that there are two aspects to any possible solution.  One is at
the OS.  Debian needs to support installing multiple versions of a package
easily, whether that's by putting the version number in the package name as I
think Robert suggests (but I don't like) or some other mechanism.  But it
shouldn't be doing things like symlinking 'foo' to 'foo2' because only the
applications themselves know what versions they are compatible with.  This is
why buildout has a configuration file that you can use to specific explicitly
the library versions you want (and let the dependency management algorithms
figure out the rest).  It also gives you an isolated environment so that you
don't care what the OS has.

So something like a configuration file that would ship with the pyfoo app that
said "give me foo-1, bar-2.3, and baz-7.0" would set up the interpreter's
run-time environment so that those package versions are mapped to "import
foo", "import bar", and "import baz".

>* What about instances where a dependent-library of consumer-of-foo1
>also wants to 'import foo' but needs foo2? Now I have to make sure
>the entire chain works with foo1. How can I do that?

IMO, you can't.

>* How do I efficiently and reliably prepend that lib-packages
>directory only when using consumer-of-foo1
>
>
>While I think Robert described it as "TERRIBLE" when I suggested
>it the other day, the way that pylons does this, I think its at
>least simple and understandable.
>
>For working on the CLI, pylons simply spawns a shell that sets
>PYTHONPATH/PYTHONHOME. Likewise, one is required to do so when
>running their particular pylons based app as a web application.
>This allows you to run easy_install or anything else, inside that
>CLI, and the libraries are installed local to the application.

I don't like that sub-shell approach much myself, but I've had to do it in a
previous life to properly select C++ shared libraries for a compiled
application.  Of course, this was on Solaris and many years ago. ;).  I
sympathize with Robert when he calls those approaches "terrible".  virtualenv,
buildout, and similar are at least less one-off hacky approaches, and the
latest version of buildout I think can optionally be told to use system Python
installed packages if they match the requested version number.

There are folks who think that *every* Python application should essentially
have its own isolated environment.  I've held the opinion that it's dangerous
to use the system Python both for essential system operations (as Debian,
Ubuntu, and Gentoo among others do), and also for deployment of applications
on those operating systems.  I've seen situations in the past that trying to
craft a proper environment for a particular app broke the OS.

>Its not necessarily ideal, but it shows the great lengths one must
>go to to have multiple versions of libraries.

Yep.  No doubt it's a crappy situation we find ourselves in today.  Why not
plan on coming to Pycon 2011 and sprinting with the distutils-sig? :)

-Barry


signature.asc
Description: PGP signature

Re: Python packaging, dependencies, upstream facilities

[Piotr Ożarowski]

[Clint Byrum, 2010-09-22]
> On Sep 21, 2010, at 3:26 AM, Piotr Ożarowski wrote:
> > but this way you cannot `import foo` anymore, you'll have to change all
> > import lines (s/foo/foo2/) even if your code is not affected by API change
> 
> Because languages like python do runtime call resolution, they
> cannot, and should not, be treated like compiled languages in this
> respect.
> 
> In python's case, if I do
> 
> import foo
> 
> x = foo.bar()
> 
> And foo has renamed bar to baz, I won't find out until I actually
> run the code that calls the missing class definition. With C++,
> there's a class missing, and at the bare minimum, I'll fail at link
> time, probably during compile.

you're right, we don't want the Ruby mess in Python

[...]
> Robert is, I think, suggesting that packaging could do this as well.
> Right now each python library can exist on the system once. If it
> conflicts, game over.
> 
> But, if you can get the order of the library loading path right,
> then this structure solves Robert's original desire:
> 
> 
> /usr/share/pyshared/foo1
> /usr/share/pyshared/foo2
> /usr/share/pyshared/foo => foo2
> /usr/share/pyshared/consumer-of-foo1/lib-packages/foo => 
> /usr/share/pyshared/foo1
> 
> 
> This would be quite easily doable in debhelper scripts, and it means
> that users can do 'import foo' in their code, but integrators can
> package things appropriately to override the "whatever version"
> with specific versions.

this approach has exactly the same problem as API in (Python) package
name. Lets say App1 requires foo and bar to work, it needs foo >=2
but bar works only with foo1. By adding consumer-of-foo1/lib-packages/
to sys.path bar is now broken.

> Off the top of my head, these are a few non-trivial issues to solve:
> 
> 
> * What about instances where a dependent-library of consumer-of-foo1
> also wants to 'import foo' but needs foo2? Now I have to make sure
> the entire chain works with foo1. How can I do that?

exactly, so we're back to educating upstreams about importance of stable
API

[...] 
> For working on the CLI, pylons simply spawns a shell that sets
> PYTHONPATH/PYTHONHOME. Likewise, one is required to do so when
> running their particular pylons based app as a web application.
> This allows you to run easy_install or anything else, inside that
> CLI, and the libraries are installed local to the application.

did you mean virtualenv?

> Its not necessarily ideal, but it shows the great lengths one must
> go to to have multiple versions of libraries.

there's also python-wxgtk2.X approach, but I don't like it either
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100922073312.gf15...@piotro.eu

Re: Python packaging, dependencies, upstream facilities

[Clint Byrum]

On Sep 21, 2010, at 3:26 AM, Piotr Ożarowski wrote:

> [Simon McVittie, 2010-09-21]
>> On Tue, 21 Sep 2010 at 10:30:33 +0200, Piotr Ożarowski wrote:
>>> I see only one sane way to fix the problem - changing Python interpreter
>>> to recognize API from filenames, like foo.1.py foo.2.py foo.2.3.py
>>> (with `import foo <= 2` as valid syntax) and let upstream authors decide
>>> when to bump it, just like C guys do, but that's a topic for
>>> python-devel mailing list...
>> 
>> If upstreams are going to do this, surely they could do so just as 
>> effectively
>> without interpreter changes, by versioning the imported module?
> 
> but this way you cannot `import foo` anymore, you'll have to change all
> import lines (s/foo/foo2/) even if your code is not affected by API change

Because languages like python do runtime call resolution, they
cannot, and should not, be treated like compiled languages in this
respect.

In python's case, if I do

import foo

x = foo.bar()

And foo has renamed bar to baz, I won't find out until I actually
run the code that calls the missing class definition. With C++,
there's a class missing, and at the bare minimum, I'll fail at link
time, probably during compile.

Because of this, I think this is at least somewhat out of band for
the interpreter, and up to the integrators and consumers of the
libraries to define the requirements of a particular set of programs.

In the java world, they use maven because it handles this for them.
They create a maven spec file that says "I need libX, libY, and
libZ (v1.1)". maven, during the build, goes out and finds libX and
libY's latest versions, then finds the closest match to libZ v1.1.
These are placed in jars in the classpath for the project, and viola
it "just works".

Of course, java also has compile time checks, so they can actually
do this with a bit more likelihood of things working (if it compiles
it works, right? ;)

Robert is, I think, suggesting that packaging could do this as well.
Right now each python library can exist on the system once. If it
conflicts, game over.

But, if you can get the order of the library loading path right,
then this structure solves Robert's original desire:


/usr/share/pyshared/foo1
/usr/share/pyshared/foo2
/usr/share/pyshared/foo => foo2
/usr/share/pyshared/consumer-of-foo1/lib-packages/foo => 
/usr/share/pyshared/foo1


This would be quite easily doable in debhelper scripts, and it means
that users can do 'import foo' in their code, but integrators can
package things appropriately to override the "whatever version"
with specific versions.

Off the top of my head, these are a few non-trivial issues to solve:


* What about instances where a dependent-library of consumer-of-foo1
also wants to 'import foo' but needs foo2? Now I have to make sure
the entire chain works with foo1. How can I do that?

* How do I efficiently and reliably prepend that lib-packages
directory only when using consumer-of-foo1


While I think Robert described it as "TERRIBLE" when I suggested
it the other day, the way that pylons does this, I think its at
least simple and understandable.

For working on the CLI, pylons simply spawns a shell that sets
PYTHONPATH/PYTHONHOME. Likewise, one is required to do so when
running their particular pylons based app as a web application.
This allows you to run easy_install or anything else, inside that
CLI, and the libraries are installed local to the application.

Its not necessarily ideal, but it shows the great lengths one must
go to to have multiple versions of libraries.


--
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8ef14727-1029-47f8-bfa4-fd11fbd30...@ubuntu.com

Re: Python packaging, dependencies, upstream facilities

[Piotr Ożarowski]

[Barry Warsaw, 2010-09-21]
> On Sep 21, 2010, at 10:30 AM, Piotr Ożarowski wrote:
> >I see only one sane way to fix the problem - changing Python
> >interpreter to recognize API from filenames, like foo.1.py foo.2.py
> >foo.2.3.py (with `import foo <= 2` as valid syntax) and let upstream
> >authors decide when to bump it, just like C guys do, but that's a
> >topic for python-devel mailing list...
> 
> It would be, but I'm skeptical.   In general, libraries tend to come in
> (Python) packages these days and you're generally not going to mix APIs in
> files within the same package, unless the package is itself managing multiple
> APIs.  It might be something that distutils2 can attack, either through its
> metadata files or by encoding the API in package directory names.

foo.1/ foo.2/ foo.2.3/ i.e. versioned (Python) packages (and not
modules) sounds fine to me.

Changes in distutils2 will not help us much, as we still have to store
files somewhere (unless we'll automatically rename top level directories
to match .dist-info's API)
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100921141921.ge15...@piotro.eu

Re: Python packaging, dependencies, upstream facilities

[Barry Warsaw]

On Sep 21, 2010, at 10:30 AM, Piotr Ożarowski wrote:

>[Robert Collins, 2010-09-20]
>> Path to a solution: use an API marker analgous to the ABI markers C
>> libraries have. Incompatible changes to a package bump the package
>> *name*. e.g.
>> python-zope.publication2.3 to python-zope.publication2.4
>> Compatible changes don't:
>> python-zope.publication2.3-2.3.0ubuntu0 to
>> python-zope.publication2.3-2.3.1ubuntu0
>
>Gustavo does something like this with CherryPy (python-cherrypy and
>python-cherrypy3 conflict with each other due to the same namespace,
>though). I maintain python-jinja and python-jinja2 but my upstream
>*did* change module name after breaking API (I will remove python-jinja
>after releasing Squeeze).

Of course, this happens all the time.  Witness unittest2, distutils2, urllib2,
httplib2 and the list goes on.  It's ugly and inconvenient, but in many ways
easier to manage and at least we *understand* how it works!

>I'm not sure we should try to solve this. IMHO we should try to
>convince upstreams that breaking API/ABI so often is a bad thing
>instead.

Good luck. :)

>Please note that:
>* 3rd party Python module authors break API or ABI a lot, and I mean
>  A LOT (sic!). Even if we'll just Provide: python-foo-API, that will
>  produce a lot of new virtual packages and a lot of confusion,
>* something as simple as changing --install-lib causes many Python
>  modules to break (hardcoded paths, things like
>  `os.path.join(os.path.dirname(__file__), '..', '..', 'share',
> 'foo')`, etc.) so playing with sys.path will require patching even
> more files,
>* adding new source package (which is the easiest way to support one
>  more upstream version at the same time) requires going thru NEW which
>  would make ftp-masters very unhappy (dozens of NEW packages every
> month),
>* maintaining 2 versions of the same module means twice as many work
>for
>  both Debian maintainer and security team
>* ...
>
>I see only one sane way to fix the problem - changing Python
>interpreter to recognize API from filenames, like foo.1.py foo.2.py
>foo.2.3.py (with `import foo <= 2` as valid syntax) and let upstream
>authors decide when to bump it, just like C guys do, but that's a
>topic for python-devel mailing list...

It would be, but I'm skeptical.   In general, libraries tend to come in
(Python) packages these days and you're generally not going to mix APIs in
files within the same package, unless the package is itself managing multiple
APIs.  It might be something that distutils2 can attack, either through its
metadata files or by encoding the API in package directory names.

-Barry



signature.asc
Description: PGP signature

Re: Fw: Python packaging, dependencies, upstream facilities

[Piotr Ożarowski]

[Simon McVittie, 2010-09-21]
> On Tue, 21 Sep 2010 at 10:30:33 +0200, Piotr Ożarowski wrote:
> > I see only one sane way to fix the problem - changing Python interpreter
> > to recognize API from filenames, like foo.1.py foo.2.py foo.2.3.py
> > (with `import foo <= 2` as valid syntax) and let upstream authors decide
> > when to bump it, just like C guys do, but that's a topic for
> > python-devel mailing list...
> 
> If upstreams are going to do this, surely they could do so just as effectively
> without interpreter changes, by versioning the imported module?

but this way you cannot `import foo` anymore, you'll have to change all
import lines (s/foo/foo2/) even if your code is not affected by API change
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100921102657.gd15...@piotro.eu

Re: Fw: Python packaging, dependencies, upstream facilities

[Simon McVittie]

On Tue, 21 Sep 2010 at 10:30:33 +0200, Piotr Ożarowski wrote:
> I'm not sure we should try to solve this. IMHO we should try to convince
> upstreams that breaking API/ABI so often is a bad thing instead.

... and that when they do, they need to rename the module with a version
number, just like C upstreams do (in Python that'd mean something like:
from foo import bar -> from foo2 import bar).


> I see only one sane way to fix the problem - changing Python interpreter
> to recognize API from filenames, like foo.1.py foo.2.py foo.2.3.py
> (with `import foo <= 2` as valid syntax) and let upstream authors decide
> when to bump it, just like C guys do, but that's a topic for
> python-devel mailing list...

If upstreams are going to do this, surely they could do so just as effectively
without interpreter changes, by versioning the imported module?

S


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/20100921095750.gb6...@reptile.pseudorandom.co.uk

Re: Fw: Python packaging, dependencies, upstream facilities

[Piotr Ożarowski]

[Piotr Ożarowski, 2010-09-21]
> [Robert Collins, 2010-09-20]
> > Path to a solution: use an API marker analgous to the ABI markers C
> > libraries have. Incompatible changes to a package bump the package
> > *name*. e.g.
> > python-zope.publication2.3 to python-zope.publication2.4
> > Compatible changes don't:
> > python-zope.publication2.3-2.3.0ubuntu0 to
> > python-zope.publication2.3-2.3.1ubuntu0

oh, I forgot to mention that we have PyDist¹ to control generated
dependencies (example: packages that build depend on python-numpy or
python-sqlalchemy will get API hardcoded in Depends if they use
dh_python{2,3} or dh_numpy) but that's not exactly what you want.

[¹] /usr/share/doc/python-doc/README.PyDist.html or
http://alioth.debian.org/scm/loggerhead/pkg-python/python-defaults-debian/annotate/head:/pydist/README.PyDist
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100921101840.gc15...@piotro.eu

Re: Fw: Python packaging, dependencies, upstream facilities

[Piotr Ożarowski]

[Robert Collins, 2010-09-20]
> Path to a solution: use an API marker analgous to the ABI markers C
> libraries have. Incompatible changes to a package bump the package
> *name*. e.g.
> python-zope.publication2.3 to python-zope.publication2.4
> Compatible changes don't:
> python-zope.publication2.3-2.3.0ubuntu0 to
> python-zope.publication2.3-2.3.1ubuntu0

Gustavo does something like this with CherryPy (python-cherrypy and
python-cherrypy3 conflict with each other due to the same namespace,
though). I maintain python-jinja and python-jinja2 but my upstream
*did* change module name after breaking API (I will remove python-jinja
after releasing Squeeze).

I'm not sure we should try to solve this. IMHO we should try to convince
upstreams that breaking API/ABI so often is a bad thing instead.

Please note that:
* 3rd party Python module authors break API or ABI a lot, and I mean
  A LOT (sic!). Even if we'll just Provide: python-foo-API, that will
  produce a lot of new virtual packages and a lot of confusion,
* something as simple as changing --install-lib causes many Python
  modules to break (hardcoded paths, things like
  `os.path.join(os.path.dirname(__file__), '..', '..', 'share', 'foo')`,
  etc.) so playing with sys.path will require patching even more files,
* adding new source package (which is the easiest way to support one
  more upstream version at the same time) requires going thru NEW which
  would make ftp-masters very unhappy (dozens of NEW packages every month),
* maintaining 2 versions of the same module means twice as many work for
  both Debian maintainer and security team
* ...

I see only one sane way to fix the problem - changing Python interpreter
to recognize API from filenames, like foo.1.py foo.2.py foo.2.3.py
(with `import foo <= 2` as valid syntax) and let upstream authors decide
when to bump it, just like C guys do, but that's a topic for
python-devel mailing list...
-- 
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl  www.griffith.cc   www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645


-- 
To UNSUBSCRIBE, email to debian-python-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100921083032.gb15...@piotro.eu

Fw: Python packaging, dependencies, upstream facilities

[Barry Warsaw]

Robert Collins has an interesting use case, though I'm not sure about his
proposed solution.  This probably touches on upstream and Debian packaging, so
in the spirit of starting a discussion, I forward his pvtmsg here for debate
(with his permission).

-Barry

Begin forwarded message:

Date: Fri, 20 Aug 2010 09:22:20 +1200
From: Robert Collins 
To: Barry Warsaw 
Subject: Python packaging, dependencies, upstream facilities


So, I'm going to give you a use case that debian packages suck at for
python (they don't for C) and how I see a path-to-a-solution.

If you were to make this happen, it would be oh so lovely.

The use case - bare essentials, there are many versions of this.
Deploy 2 versions of Launchpad to the same host at the same time, when
an incompatible change to a dependency has happened.

E.g. zope does a release that moves an attribute; our old version of
LP would no longer work, and our new version of LP has other changes
that mean it won't work on the old zope.


How does this suck? If you upgrade the old zope, the running process
is now running without the python files it loaded. Ugh.
If you don't upgrade the old zope, you can't run the new LP. Ugh.


What we do today, you know: we don't use debian packages for the
dependencies. Which means we need a whole parallel dependency
structure, tools to manage them, etc etc.


Path to a solution: use an API marker analgous to the ABI markers C
libraries have. Incompatible changes to a package bump the package
*name*. e.g.
python-zope.publication2.3 to python-zope.publication2.4
Compatible changes don't:
python-zope.publication2.3-2.3.0ubuntu0 to
python-zope.publication2.3-2.3.1ubuntu0


Both packages would provide python-zope.publication, but not conflict
with each other. Software built on these that needs a specific version
of the API would depend on python-zope.publication2.4 or whatever. And
a debhelper script, like the C ABI one, could look at what build deps
are installed to automate *that*.


Problem is, stock python, has no API versioning concept. 'import foo'
-> there can only be one. So one needs someline like 'Require(2.3)' or
however the black magic setuptools stuff is spelt to be available.
Specifically:
 - python-packageN and python-packageN+Y must be coinstallable
 - neither can use /site-packages/package
 - importing just 'package' should get the highest versioned one
unless one is already loaded
 - using Require or something should get the highest versioned one
matching your needs unless one is already loaded.


Get this working, and the use case becomes:
Install both versions of the dependency.
Profit

-Rob


signature.asc
Description: PGP signature