On Tue, Mar 5, 2013 at 10:27 PM, Mark McLoughlin <mar...@redhat.com> wrote: > Ok, there's a tonne of details there about pip, virtualenv and .pth > files that are going over my head right now, but the general idea I'm > taking away is: > > - the system has multiple versions of somedep installed under /usr > somewhere > > - the latest version (2.1) is what you get if you naively just do > 'import somedep' > > - most applications should instead somehow explicitly say they need > ~>1.3, ->1.6 or ~->2.0 or whatever > > - distros would have multiple copies of the same library, but only > one copy for each incompatible stream rather than one copy for each > application > > That's definitely workable.
That's the general idea. There's a fair bit of work needed to get there though, and the on-disk layouts for the distros would look fairly different from the current approach of just dumping everything in Python's site-packages directory. It also won't solve *all* compatibility problems, so the distros may still be on the hook to carry some compatibility patches in order to get a common dependency that works for everyone. (To be honest, I expect software collections will end up needing a similar capability, for all the reasons you gave earlier in response to the suggestion of just using virtualenv with the existing simple bundling approach) >> If any distros want that kind of thing to become a reality, though, >> they're going to have to step up and contribute it. As noted above, >> for the current tool development teams, the focus is on distributing, >> maintaining and deploying cross-platform applications, not making it >> easy to do security updates on a Linux distro. I believe it's possible >> to satisfy both parties, but it's going to be up to the distros to >> offer a viable plan for meeting their needs without disrupting >> existing upstream practices. > > Point well taken. > > However, I am surprised the pendulum has swung such that Python platform > maintainers only worry about app maintainers and not distro maintainers. If you saw my rant on the Fedora python-devel list, you may have guessed that a lot of the expressed frustration with the distro approach and the lack of willingness to understand why bundling is such an attractive option for cross-platform development comes from me personally, rather than the Python community in general. However, there's also a general dearth of distro people on the upstream packaging lists to balance the web app developers - I'm one of the ones that is *most sympathetic* to the distro point of view, and that should worry you a bit. > And also, Python embracing incompatible updates and bundling is either a > new upstream practice or just that doesn't well understood on the distro > side. The popularity of bundling is primarily in devops and integrated application stacks. You can't deploy on Windows without bundling, and when you have to manage your own security updates for other platforms anyway, the distros refusing to accommodate that model is just frustrating. The distro packagers say "we don't trust the developers of all of the apps we ship to provide timely security fixes" and the app developers say "we don't trust the packagers of all of the distros we support to provide timely security fixes" and everybody gets annoyed with each other because their concerns and priorities are so wildly divergent. And we're definitely *not* embracing incompatible updates, we're acknowledging their inevitable existence in volunteer driven projects, and defining a way to at least communicate them clearly when they happen. >> I will note that making this kind of idea more feasible is one of the >> reasons I am making "compatible release" the *default* in PEP 426 >> version specifiers, but it still needs people to actually figure out >> the details and write the code. > > I still think that going down this road without the parallel installs > issue solved is a dangerous move for Python. Leaving aside pain for > distros for a moment, there was a perception (perhaps misguided) that > Python was a more stable platform that e.g. Ruby or Java. > > If Python library maintainers will see PEP426 as a license to make > incompatible changes more often so long as they bump their major number, > then that perception will change. This isn't a concern I had considered, but I'll review the wording in the PEP with that in mind (there's a bunch of other stuff I need to fix in the version scheme description anyway). > By bundling, I mean that an app sees itself as in control of the > versions of its dependencies. The app developer fundamentally thinks she > is delivering a specific stack of dependencies and her application code > on top rather than installing just their app and running it on a stable > platform. I'm actually trying to push *against* that by making the "compatible release" specifier the default in PEP 426 rather than requiring a separate operator (if you look at PEP 345, the predecessor that defines metadata 1.2, it pinned dependencies by default). Actual pinning (with "==") should be limited to devops type situations, where an application really is explicitly controlling it's entire stack, but that's not the kind of metadata anyone should be publishing on PyPI. I can see how the perception of the current PEP might be different without that background of knowing precisely what the previous version said, though. > Anyway, sounds like we have some ideas for parallel installs we can > investigate. Yes, I'm definitely not opposed to the idea of parallel installs - I'm just opposed to the idea of parallel install systems that rely on changes to PyPI packages in order for them to work properly. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig