On Fri, Dec 2, 2016 at 4:33 PM, Robert T. McGibbon <rmcgi...@gmail.com> wrote:
> Isn't this issue already solved by (and the raison d'être of) the multiple > third-party Python redistributors, like the various OS package maintainers, > Continuum's Anaconda, Enthought Canopy, ActiveState Python, WinPython, etc? > My intention is not creating yet another distribution. Instead, I want to see if there is interest in the different distributions on sharing some of the burden of curating by bringing up this discussion and seeing what is needed. These distributions have their recipes that allow them to build their packages using their tooling. What I propose is having some of that data community managed so the distributions can use that along with their tooling to build the eventual packages. On Fri, Dec 2, 2016 at 5:23 PM, Nick Coghlan <ncogh...@gmail.com> wrote: > On 3 December 2016 at 01:33, Robert T. McGibbon <rmcgi...@gmail.com> > wrote: > >> Isn't this issue already solved by (and the raison d'être of) the >> multiple third-party Python redistributors, like the various OS package >> maintainers, Continuum's Anaconda, Enthought Canopy, ActiveState Python, >> WinPython, etc? >> > > Yep. Once you start talking content curation, you're in a situation where: > > - you're providing an ongoing service that will always be needed (the > specific packages will change, but the task won't) > - you need to commit to a certain level of responsiveness for security > issues > - you'll generally need to span multiple language ecosystems, not just > Python > - exactly which packages are interesting will depend on the user audience > you're targeting > - the tolerance for API breakage will also vary based on the audience > you're targeting > - you'll often want to be able to carry patches that aren't present in the > upstream components > - you'll need to decide which target platforms you want to support > These are interesting issues you bring up here. What I seek is having a set that has per package a version, source, Python dependencies and build system. Other dependencies would be for now left out, unless someone has a good idea how to include those. Distributions can take this curated set and extend the data with their distribution specific things. For example, in Nix we could load such a set, map a function that builds the packages in the set, and override what is passed to the function when necessary (e.g. to add system dependencies, our patches, or how tests are invoked, and so on). Responsiveness is indeed an interesting issue. If there's enough backing, then I imagine security issues will be resolved as fast as they are nowadays by the distributions backing the initiative. > If a curation community *isn't* doing any of those things, then it isn't > adding a lot of value beyond folks just doing DIY integration in their CI > system by pinning their dependencies to particular versions. > > I would imagine that distributions that would support this idea would have a CI tracking packages built using the curated set and the distribution-specific changes. When there's an issue they could fix it at their side, or if it is something that might belong in the curated set, they would report the issue. At some point, when they would freeze, they would pin to a certain YYYY.MM and API breakage should not occur. > As far as the comments about determining dependencies goes, the way pip > does it generally works fine, you just need a sandboxed environment to do > the execution, and both redistributors and open source information > providers like libraries.io are actively working on automating that > process (coping with packages as they exist on PyPI today, rather than > relying on the upstream community to change anything about the way Python > packaging works). > > libraries.io is a very interesting initiative. It seems they scan the contents of the archives and extract dependencies based on what is in the requirements files, which is often more than is actually needed for building and running the package. They would benefit from having a declarative style for the dependencies and build system, but that is another issue (PEP 517 e.g.) than what I bring up here. We also have a tool that runs pip in a sandbox to determine the dependencies, and then provide us with an expression. It works, but it shouldn't be necessary. Freddy
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
