On Tue, Mar 5, 2013 at 8:29 AM, Mark McLoughlin <mar...@redhat.com> wrote: > On Mon, 2013-03-04 at 12:44 -0500, Donald Stufft wrote: >> On Monday, March 4, 2013 at 12:36 PM, Mark McLoughlin wrote: > >> > If parallel incompatible installs is a hopeless problem in Python, >> > why >> > the push to semantic versioning then rather than saying that >> > incompatible API changes should mean a name change? >> Forcing a name change feels ugly as all hell. I don't really see what >> parallel installs has much to do with anything. I don't bundle anything >> and i'm ideologically opposed to it generally but I don't typically have >> a need for parallel installs because I use virtual environments. Why >> don't you utilize those? (Not being snarky, actually curious). > > It's a fair question. > > To answer it with a question, how do you imagine Linux distributions > using virtual environments such that: > > $> yum install -y openstack-nova > > uses a virtual environment? How does it differ from bundling? (Not being > snarky, actually curious :) > > The approach that some Fedora folks are trying out is called "Software > Collections". It's not Python specific, but it's basically the same as a > virtual environment. > > For OpenStack, I think we'd probably have all the Python libraries we > require installed under e.g. /opt/rh/openstack-$version so that you > could have programs from two different releases of OpenStack installed > on the same system. > > Long time packagers are usually horrified at this idea e.g. > > > http://lists.fedoraproject.org/pipermail/devel/2012-December/thread.html#174872
Yes, it's the eternal tension between "I only care about making a wide variety of applications on as easy to maintain on platform X as possible" view of the sysadmin and the "I only care about making application Y as easy to maintain on a wide variety of platforms as possible" view of the developer. Windows, Android, Mac OS X, etc, pretty much dial their software distribution model all the way towards the developer end of the spectrum. Linux distro maintainers need to realise that the language communities are almost entirely down the developer end of this spectrum, where sustainable cross-platform support is much higher priority than making life easier for administrators for any given platform. We're willing to work with distros to make deployment of security updates easier, but any proposals that involve people voluntarily making cross-platform development harder simpler aren't going to be accepted. > - How many of these 200 new packages are essentially duplicates? Once > you go down the route of having applications bundle libraries like > this, there's going to basically be no sharing. There's no sharing only if you *actually* bundle the dependencies into each virtualenv. While full bundling is the only mode pip currently implements, completely isolating each virtualenv, it doesn't *have* to work that way. In particular, PEP 426 offers the opportunity to add a "compatible release" mode to pip/virtualenv where the tool can maintain a shared pool of installed libraries, and use *.pth files to make an appropriate version available in each venv. Updating the shared version to a more recent release would then automatically update any venvs with a *.pth file that reference that release. For example, suppose an application requires "somedep (1.3)". This requires at least version 1.3, and won't accept 2.0. The latest available qualifying version might be "1.5.3". At the moment, pip will install a *copy* of somedep 1.5.3 into the application's virtualenv. However, it doesn't have to do that. It could, instead, install somedep 1.5.3 into a location like "/usr/lib/shared/pip-python/somedep1/<contents>", and then add a "somedep1.pth" file to the virtualenv that references "/usr/lib/shared/pip-python/somedep1/". Now, suppose we install another app, also using a virtualenv, that requires "somedep (1.6)". The new version 1.6.0 is available now, so we install it into the shared location and *both* applications will end up using somedep 1.6.0. A security update is released for "somedep" as 1.6.1 - we install it into the shared location, and now both applications are using 1.6.1 instead of 1.6.0. Yay, that's what we wanted, just as if we had runtime version selection, only the selection happens at install time (when adding the *.pth file to the virtualenv) rather than at application startup. Finally, we install a third application that needs "somedep (2.1)". We can't overwrite the shared version, because it isn't compatible. Fortunately, what we can do instead is install it to "/usr/lib/shared/pip-python/somedep2/<contents>" and create a "somedep2.pth" file in that environment. The two virtualenvs relying on "somedep1" are blissfully unaware anything has changed because that version never appears anywhere on their sys.path. Could you use this approach for the actual system site-packages directory? No, because sys.path would become insanely long with that many *.pth files. However, you could likely symlink to releases stored in the *.pth friendly distribution store. But for application specific virtual environments, it should be fine. If any distros want that kind of thing to become a reality, though, they're going to have to step up and contribute it. As noted above, for the current tool development teams, the focus is on distributing, maintaining and deploying cross-platform applications, not making it easy to do security updates on a Linux distro. I believe it's possible to satisfy both parties, but it's going to be up to the distros to offer a viable plan for meeting their needs without disrupting existing upstream practices. I will note that making this kind of idea more feasible is one of the reasons I am making "compatible release" the *default* in PEP 426 version specifiers, but it still needs people to actually figure out the details and write the code. I will also note that the filesystem layout described above is *far* more amenable to safe runtime selection of packages than the current pkg_resources method. The critical failure mode in pkg_resources that can lead to a lot of weirdness is that it can end up pushing site-packages itself on to the front of sys.path which can shadow a *lot* of modules (in particular, an installed copy of the software you're currently working on may shadow the version in your source checkout - this is the bug the patch I linked earlier was needed to resolve). Runtime selection would need more work than getting virtual environments to work that way, but it's certainly feasible once the installation layout is updated. > - What's the chance that that all of these 200 packages will be kept > up to date? If an application works with a given version of a > library and it can stick with that version, it will. As a Python > library maintainer, wow do you like the idea of 10 different > versions of you library included in Fedora? That's a problem the distros need to manage by offering patches to how virtual environments and installation layouts work, rather than lamenting the fact that cross-platform developers and distro maintainers care about different things. > - The next time a security issue is found in a common Python library, > does Fedora now have to rush out 10 parallel fixes for it? Not if Fedora contributes the changes needed to support parallel installs without requiring changes to existing Python applications and libraries. > You can see that reaction in mails like this: > > http://lists.fedoraproject.org/pipermail/devel/2012-December/174944.html > > and the "why can't these losers just maintain compatibility" view: > > http://lists.fedoraproject.org/pipermail/devel/2012-December/175028.html > http://lists.fedoraproject.org/pipermail/devel/2012-December/174929.html > > Notice folks complaining about Ruby and Java here, not Python. I can see > Python embracing semantic versioning and "just use venv" shortly leading > to Python being included in the list of "heretics". Unlike Java, the Python community generally sees *actual* bundling as evil - expressing constraints relative to a published package index is a different thing. Dependencies in Python are typically only brought together into a cohesive, pinned set of version by application developers and system integrators - the frameworks and libraries often express quite loose version requirements (and receive complaints if they're overly restrictive). The distros just have a harder problem than most because the set of packages they're trying to bring together is so large, they're bound to run into many cases of packages that have mutually incompatible dependencies. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
