> Maybe it would help if you have a concrete example of a scenario where they would benefit from having this distinction?
In the Nix package manager (source distribution with binary substitutes) and Nixpkgs package set we typically require the filename and hash of a package. In our expressions we typically pass an url (that includes the name), and the hash. The url is only needed when the file isn't in our store. This is convenient, because if an url is optional this allows you to pre-fetch or work with mirrors. All we care about is that we get the file, not how it is provided. This applies for source archives, but behind the scenes also for binary substitutes. With Nix, functions build a package, and dependencies are passed as function arguments with names that typically, but not necessarily, resemble the dependency name. Now, a function that builds a package, a package builder, only needs to be provided with abstract dependencies; it just needs to know what it should look for, "we need 'a' numpy, 'a' scipy, 'a compiler that has a certain interface and can do this job'", etc.. Version numbers can help in order to fail prematurely, but generally only bounds, not a pinned value. Its up to another tool to provide the builder with the actual packages, the concrete dependencies to the builder. And this tool might fetch it from PyPI, or from GitHub, or... The same goes for building, distributing and installing Python packages. Setuptools shouldn't bother with versions (except the constraints in case of libraries) or wherever a source comes from but just build or fail. Pip should just fetch/resolve and pass concrete dependencies to whatever builder (Setuptools, Flit), or whatever environment (virtualenv) needs it. It's quite frustrating as a downstream having to deal with packages where versions are pinned unnecessarily and therefore I've also requested on the Setuptools tracker a flag that ignores constraints [1] (though I fear I would have to pull up my sleeves for this one :) ) . [1] https://github.com/pypa/setuptools/issues/894 On Wed, Feb 15, 2017 at 3:11 PM, Nathaniel Smith <n...@pobox.com> wrote: > On Wed, Feb 15, 2017 at 5:27 AM, Nick Coghlan <ncogh...@gmail.com> wrote: > > On 15 February 2017 at 12:58, Nathaniel Smith <n...@pobox.com> wrote: > >> On Wed, Feb 15, 2017 at 3:33 AM, Nick Coghlan <ncogh...@gmail.com> > wrote: > >>> - "requires": list where entries are either a string containing a PEP > >>> 508 dependency specifier or else a hash map contain a "requires" key > >>> plus "extra" or "environment" fields as qualifiers > >>> - "integrates": replacement for "meta_requires" that only allows > >>> pinned dependencies (i.e. hash maps with "name" & "version" fields, or > >>> direct URL references, rather than a general PEP 508 specifier as a > >>> string) > >> > >> What's accomplished by separating these? I really think we should > >> strive to have fewer more orthogonal concepts whenever possible... > > > > It's mainly a matter of incorporating > > https://caremad.io/posts/2013/07/setup-vs-requirement/ into the core > > data model, as this distinction between abstract development > > dependencies and concrete deployment dependencies is incredibly > > important for any scenario that involves > > publisher-redistributor-consumer chains, but is entirely non-obvious > > to folks that are only familiar with the publisher-consumer case that > > comes up during development-for-personal-and-open-source-use. > > Maybe I'm just being dense but, umm. I don't know what any of these > words mean :-). I'm not unfamiliar with redistributors; part of my > confusion is that this is a concept that AFAIK distro package systems > don't have. Maybe it would help if you have a concrete example of a > scenario where they would benefit from having this distinction? > > > One particular area where this is problematic is in the widespread > > advice "always pin your dependencies" which is usually presented > > without the all important "for application or service deployment" > > qualifier. As a first approximation: > > pinning-for-app-or-service-deployment == good, > > pinning-for-local-testing == good, > > pinning-for-library-or-framework-publication-to-PyPI == bad. > > > > pipenv borrows the Ruby solution to modeling this by having Pipfile > > for abstract dependency declarations and Pipfile.lock for concrete > > integration testing ones, so the idea here is to propagate that model > > to pydist.json by separating the "requires" field with abstract > > development dependencies from the "integrates" field with concrete > > deployment dependencies. > > What's the benefit of putting this in pydist.json? I feel like for the > usual deployment cases (a) going straight from Pipfile.lock -> venv is > pretty much sufficient, with no need to put this into a package, but > (b) if you really do want to put it into a package, then the natural > approach would be to make an empty wheel like > "my-django-app-deploy.whl" whose dependencies were the contents of > Pipfile.lock. > > There's certainly a distinction to be made between the abstract > dependencies and the exact locked dependencies, but to me the natural > way to model that distinction is by re-using the distinction we > already have been source packages and binary packages. The build > process for this placeholder wheel is to "compile down" the abstract > dependencies into concrete dependencies, and the resulting wheel > encodes the result of this compilation. Again, no new concepts needed. > > > In the vast majority of publication-to-PyPi cases people won't need > > the "integrates" field, since what they're publishing on PyPI will > > just be their abstract dependencies, and any warning against using > > "==" will recommend using "~=" or ">=" instead. But there *are* > > legitimate uses of pinning-for-publication (like the PyObjC > > metapackage bundling all its subcomponents, or when building for > > private deployment infastructure), so there needs to be a way to > > represent "Yes, I'm pinning this dependency for publication, and I'm > > aware of the significance of doing so" > > Why can't PyObjC just use regular dependencies? That's what distro > metapackages have done for decades, right? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig >
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
