Disclaimer: everything I say below about pip is ultimately up to the pip devs. I'm just pointing out what I think makes sense, and my reading of Donald's comments means that I expect he would feel the same way.
On 22 August 2013 17:22, Vinay Sajip <vinay_sa...@yahoo.co.uk> wrote: > Nick Coghlan <ncoghlan <at> gmail.com> writes: >> What that means though, is we need *something else* that indicates the >> common core that people can assume will always be available. It's this > > If that "something else" you're thinking of is something that is supposed to > live in the stdlib, then I see no reason why a subset of distlib couldn't be > that something else, since stdlib changes are not on the table for 3.4. I > certainly have never envisaged that distlib would be adopted wholesale into > the stdlib (if at all) without peer review and any changes coming out of that. I *can't* tell people "we're going to migrate to distlib as the reference packaging infrastructure implementation" when I really mean "we're going to migrate to some as yet undefined subset of distlib, so 'import distlib' won't be the long term answer". Incorporating only a subset of an existing published API into the standard library is a mistake - the PyXML debacle shows us that. If the API is different (even if that means a strict subset), then it needs a different name. >> common core which pip will need to factor out to remove their >> dependency on setuptools, rather than adopting distlib wholesale, >> experimental features and all. > > I honestly think you're making a bit too much of the "experimental" label > here, even though it is a label that I use myself. For me, that label is > most appropriate for the extended metadata that I collect from PyPI and > which is the basis for distlib's smarter dependency resolution. > > If your concerns are about instability due to experimental features (and I > quite understand the importance of stability in packaging), then there's > nothing stopping anyone doing a technical review of distlib to see what any > actual risks are. Indeed, I've invited such review from day one. It has nothing to do with code quality, and everything to do with being able to explain the migration plan to people. I *can* say to them "pip is going to cherry pick parts of distlib and potentially other libraries and make them available as 'piplib', which will be installed automatically when you install pip". At the moment, I no longer feel I can say "distlib will become the reference implementation". Note that there's also the bootstrapping issue with having pip depend on an external library: having the core library *in pip* makes that problem go away. >> Currently, pip doesn't expose a programmatic API. I suggested to >> Donald that it may make sense to start exposing one as "piplib". The > > I think this would be a mistake, and it seems a little early to make this > sort of decision. You've given me to understand that pip could at some > future point use (some subset of) distlib under the covers, with > compatibility maintained at the CLI level. If that is still the case, then I > don't see much value in having two lib layers. When I made that suggestion, I misunderstood your plans for distlib. If pip are only adopting a subset of it, they can't use the same name, or people will get confused. I now think it makes more sense for pip to migrate to a more tightly constrained public library API, that doesn't go beyond the documented metadata standards (except for legacy compatibility reasons). > Like setuptools, pip has done sterling service, but it's not a codebase I'd > like to see become the basis for our long-term packaging infrastructure. I > don't mean to offend anybody by saying that - it's just software that has > grown organically over time and IMO there will be technical debt to repay if > we go down the route of exposing bits of it as Python APIs. I don't expect the contents of piplib to match the contents of the existing pip module. This is about enabling a gradual refactoring over to a cleaner core library with a public API, rather than a big bang migration to an alternative solution like distlib. > It certainly feels like you're side-lining distlib, or planning to, whether > or not that's the message you're intending to send. No matter :-) I don't currently believe your plans for distlib and my plans for the standard library software distribution support (whether directly in the standard library or via the pip bootstrapping) are compatible. If I am correct, then distlib remains extemely valuable in that scenario, but the nature of its role changes. I want a completely barebones "absolutely no features that aren't covered by an Accepted PEP" library as a candidate for future inclusion. Such a library *cannot* be particularly useful at this point in time, because the metadata 2.0 PEPs are nowhere near complete enough for that. By contrast, you understandably wish for distlib to be useful *now*, which means running ahead of the standardisation process, filling in missing features as needed. Assuming the pip devs are amenable (and given Donald's comments in this thread, I expect they will be), making the decision now that pip *will not* bundle distlib directly, but instead will create its own support library means we have a clear path forward for defining the "suitable subset of distlib (and perhaps other libraries)" that will become the "available by default" core library for installation tools, while leaving you free to make distlib as useful as it can be in the near term, even if it takes the standardisation process a while to catch up. The eventual core API *probably won't support* the legacy metadata formats, instead leaving that to setuptools and/or distlib. It will probably depend on the upgraded PyPI APIs as well (once they're defined). Ideally, this will give us at least two competing implementations on the metadata 2.0 production side (setuptools/bdist_wheel and distlib/distil) and two on the consumption side (piplib/pip and distlib/distil), so the various combinations should help us ensure we've eliminated most of the ambiguity from the specifications and aren't going to the end up with excessively high levels of implementation defined behaviour again. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia On 22 August 2013 17:22, Vinay Sajip <vinay_sa...@yahoo.co.uk> wrote: > Nick Coghlan <ncoghlan <at> gmail.com> writes: > >> I previously thought distlib was going to be the repository for the >> agreed, stable, "this is going to happen" stuff. It's OK that I was >> wrong - I think you're right that somewhere is needed as an >> experimental location to show some of the *possibilities* of the new >> metadata, and to seed ideas for making it into the eventual standard >> base that people can assume is readily available. > > It's not just about completely new, experimental stuff. For example, the > resources functionality isn't completely new territory. The PyPI interfacing > is (IMO) a saner API than the one in distutils2. A better Windows story (for > when launcher support when py.exe can't be used) is also not rocket science. > >> What that means though, is we need *something else* that indicates the >> common core that people can assume will always be available. It's this > > If that "something else" you're thinking of is something that is supposed to > live in the stdlib, then I see no reason why a subset of distlib couldn't be > that something else, since stdlib changes are not on the table for 3.4. I > certainly have never envisaged that distlib would be adopted wholesale into > the stdlib (if at all) without peer review and any changes coming out of that. > >> common core which pip will need to factor out to remove their >> dependency on setuptools, rather than adopting distlib wholesale, >> experimental features and all. > > I honestly think you're making a bit too much of the "experimental" label > here, even though it is a label that I use myself. For me, that label is > most appropriate for the extended metadata that I collect from PyPI and > which is the basis for distlib's smarter dependency resolution. > > If your concerns are about instability due to experimental features (and I > quite understand the importance of stability in packaging), then there's > nothing stopping anyone doing a technical review of distlib to see what any > actual risks are. Indeed, I've invited such review from day one. > >> Currently, pip doesn't expose a programmatic API. I suggested to >> Donald that it may make sense to start exposing one as "piplib". The > > I think this would be a mistake, and it seems a little early to make this > sort of decision. You've given me to understand that pip could at some > future point use (some subset of) distlib under the covers, with > compatibility maintained at the CLI level. If that is still the case, then I > don't see much value in having two lib layers. > > Like setuptools, pip has done sterling service, but it's not a codebase I'd > like to see become the basis for our long-term packaging infrastructure. I > don't mean to offend anybody by saying that - it's just software that has > grown organically over time and IMO there will be technical debt to repay if > we go down the route of exposing bits of it as Python APIs. > > It certainly feels like you're side-lining distlib, or planning to, whether > or not that's the message you're intending to send. No matter :-) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
