On Fri, 2 Oct 2015 at 05:08 Donald Stufft <don...@stufft.io> wrote: > On October 2, 2015 at 12:54:03 AM, Nathaniel Smith (n...@pobox.com) wrote: > > > We realized that actually as far as we could tell, it wouldn't > > be that > > hard at this point to clean up how sdists work so that it would be > > possible to migrate away from distutils. So we wrote up a little > > draft > > proposal. > > > > The main question is, does this approach seem sound? > > I've just read over your proposal, but I've also just woken up so I might > be > a little slow still! After reading what you have, I don't think that this > proposal is the right way to go about improving sdists. > > The first thing that immediately stood out to me, is that it's recommending > that downstream redistributors like Debian, Fedora, etc utilize Wheels > instead > of the sdist to build their packages from. However, that is not really > going to > fly with most (all?) of the downstream redistributors. Debian for instance > has > policy that requires the use of building all of it's packages from Source, > not > from anything else and Wheels are not a source package. While it can > theoretically work for pure python packages, it quickly devolves into a > mess > when you factor in packages that have any C code what so ever. >
So wouldn't they then download the sdist, build a wheel as an intermediate, and then generate the .deb file? I mean as long as people upload an sdist for those that want to build from source and a wheel for convenience -- which is probably what most people providing wheels do anyway -- then I don't see the problem. > > Overall, this feels more like a sidegrade than an upgrade. One major theme > throughout of the PEP is that we're going to push to rely heavily on > wheels as > the primary format of installation. While that works well for things like > Debian, I don't think it's going to work as wheel for us. If we were only > distributing pure python packages, then yes absolutely, however given that > we > are not, we have to worry about ABI issues. Given that there is so many > different environments that a particular package might be installed into, > all > with different ABIs we have to assume that installing from source is still > going to be a primary path for end users to install and that we are never > going > to have a world where we can assume a Wheel in a repository. > > One of the problems with the current system, is that we have no mechanism > by > which to determine dependencies of a source distribution without > downloading > the file and executing some potentially untrusted code. This makes > dependency > resolution harder and much much slower than if we could read that > information > statically from a source distribution. This PEP doesn't offer anything in > the > way of solving this problem. > Isn't that what the requirements and requirements-file fields in the _pypackage file provide? Only if you use that requirements-dynamic would it require execcuting arbitrary code to gather dependency information, or am I missing something? > > To a similar tune, this PEP also doesn't make it possible to really get at > any other metadata without executing software. This makes it pratically > impossible to safely inspect an unknown or untrusted package to determine > what > it is and to get information about it. Right now PyPI relies on the > uploading > tool to send that information alongside of the file it is uploading, but > honestly what it should be doing is extracting that information from > within the > file. This is sort of possible right now since distutils and setuptools > both > create a static metadata file within the source distribution, but we don't > rely > on that within PyPI because that information may or may not be accurate > and may > or may not exist. However the twine uploading tool *does* rely on that, and > this PEP would break the ability for twine to upload a package without > executing arbitrary code. > Isn't that only if you use the dynamic fields? > > Overall, I don't think that this really solves most of the foundational > problems with the current format. Largely it feels that what it achieves is > shuffling around some logic (you need to create a hook that you reference > from > within a .cfg file instead of creating a setuptools extension or so) but > without fixing most of the problems. The largest benefit I see to > switching to > this right now is that it would enable us to have build time dependencies > that > were controlled by pip rather than installed implicitly via the execution > of > the setup.py. That doesn't feel like a big enough benefit to me to do a > mass > shakeup of what we recommend and tell people to do. Having people adjust > and > change and do something new requires effort, and we need something to > justify > that effort to other people and I don't think that this PEP has something > we > can really use to justify that effort. > >From my naive perspective this proposal seems to help push forward a decoupling of building using distutils/setuptools as the only way you can properly build Python projects (which is what I think we are all after) and will hopefully eventually free pip up to simply do orchestration. > > I *do* think that there is a core of some ideas here that are valuable, > and in > fact are similar to some ideas I've had. The main flaw I see here is that > it > doesn't really fix sdists, it takes a solution that would work for VCS > checkouts and then reuses it for sdists. In my mind, the supported flow for > package installation would be: > > VCS/Bare Directory -> Source Distribution -> Wheel > > This would (eventually) be the only path that was supported for > installation > but you could "enter" the path at any stage. For example, if there is a > Wheel > already available, then you jump right on at the end and just install > that, if > there is a sdist available then pip first builds it into a wheel and then > installs that, etc. > > I think your PEP is something like what the VCS/Bare Directory to sdist > tooling > could look like, but I don't think it's what the sdist to wheel path should > look like. > Is there another proposal I'm unaware for the sdist -> wheel step that is build tool-agnostic? I'm all for going with the best solution but there has to be an actual alternative to compare against and I don't know of any others right now and this proposal does seem to move things forward in a reasonable fashion.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
