On Sat, Oct 3, 2015 at 10:50 AM, Donald Stufft <don...@stufft.io> wrote: > I feel like you have some sort of "a sdist is jsut a tarball of a VCS" > mentality and I don't think that idea of a sdist is generally useful.
Hmm, so, between thinking this over more on the plane today and reading your and Paul's great replies, I think I see where a lot of this disjunction might be arising. I'll try to reply to those in more detail later, but first let me try to lay this out and see if it makes things clearer. First, let's drop the word "sdist", it's confusing. I'm starting from the long history and conventions around how people make what I'll call "source releases" (and in a few paragraphs will contrast with "source wheels"). 'Everyone knows' that when you release a new version of some package (in pretty much any language), then one key step is to put together a file called <package>-<version>.<zip or .tar.gz>. And 'everyone knows' that if you see a file that follows this naming convention, and you download it, then what you'll find inside is: a single directory called <package>-<version>/, and inside this directory will be something that's almost like a VCS checkout -- it'll probably contain a README, source files in convenient places to be edited or grepped, etc. The differences from a VCS checkout (if any) will be little convenience stuff -- like ./autogen.sh will have been run already, or there will be an extra file containing a fixed version number instead of it being autodetected, or -DNDEBUG will be in the default CFLAGS, or Cython files will have been pre-translated to C -- but fundamentally it will be similar to a VCS checkout, and switching back and forth between them won't be too jarring. 95% of the time there will be a standard way to build the thing ('./configure && make && make install', or 'python setup.py install', or similar). And these kind of source releases have a rich ecosystem around them and serve a wide range of uses: they provide a low-tech archival record (while VCS's come and go), they end up in deb and rpm "original source" bundles, they get downloaded by users and built by hand (maybe with weird configury on top, like a hack to enable cross-compilation) or poked around in by hand, etc. etc. When sdists were originally designed, then "source releases" is what the designers were thinking about. Then, easy_install came along, and pulled off a clever hack where when you asked for some package by name, then it would try to automagically go out and track down any relevant source releases and build them all. And it works great, except when it doesn't. And creates massive headaches for everyone trying to work on python packaging afterwards, because source releases were not designed to be used this way. My hypothesis is that the requirements that were confusing me are based around the idea that an sdist should be something designed to slot into this particular use case: i.e., something that pip can automatically grab and work with while solving a dependency resolution problem. Therefore it really needs to have a static name, and static version number, and static dependencies, and must produce exactly one binary wheel that shares all that metadata. Let's call this a "source wheel" -- what we're really looking for here is a way to ship extension modules inside something that acts like an architecture-neutral "none-any" wheel. So: the email that started this thread was a proposal for how to standardize the format of "source releases", and Donald's counter was a proposal for how to standardize the format of "source wheels". Does that sound right? If so, then some follow-up thoughts: 1) If we design a source wheel format, then I am 100% in favor of the suggestion of giving it a unique extension like "swhl". I'm still a young whippersnapper compared to some, but I've been downloading files named <package>-<version>.<zip or tar.gz> for 20 years, and AFAICR every one of those files unpacked to make a single directory that was laid out like a VCS checkout. Obviously we can change what goes inside, but we should change the naming convention at the same time because otherwise we're just going to confuse people. 2) I think there's a strong case to be made that Python actually needs standards for *both* source releases and source wheels. There's certainly no logical contradiction -- they're conceptually different things. It sounds like we all agree that "pip" should continue to have a way to build and install an arbitrary VCS checkout, and extending that standard to cover building and installing a classic "source release" would be... almost difficult *not* to do. And I think that there will continue to be a clear need for source releases even in a world where source wheels exist, because of all those places where source releases get used that aren't automagic-easy_install/pip-builds. For example, most pure Python packages (the ones that already make "none-any" wheels) have no need at all for source wheels, but they still need source releases to serve as archival snapshots. And more complex packages that need build-time configuration (e.g. numpy) will continue to require source releases that can be configured to build wheels that have a variety of different properties (e.g., different dependency metadata), so they can't get by with source wheels alone -- but you can imagine that such projects might reasonably *in addition* provide a source wheel that locks down the same default configuration that gets used for their uploaded binary wheel builds, and is designed for pip to use when trying to resolve dependencies on platforms where a regular binary wheel is unavailable. Pictorially, this world would look like: VCS checkout -> source release \ \ --------------------+--> in-place install | +--> wheels -> install | +--> source wheels -> wheels -> install 3) It sounds like we all agree that - 'pip install <VCS checkout>' should work - that there is some crucial metadata that VCS checkouts won't be able to provide without running arbitrary code (e.g. dependencies and version numbers) - that what metadata they do provide (e.g., which arbitrary code to run) should be specified in a human-friendly configuration file Given this, in the big picture it sounds like the only really essentially controversial things about the original proposal might be: - that 'pip install <tarball of VCS checkout>' should work the same as 'pip install <VCS checkout>' (does anyone actually disagree?) - the 1 paragraph describing the "transitional plan" allowing pip to automatically install from these source releases *as part of a dependency resolution plan* (as opposed to when a VCS-checkout-or-equivalent is explicitly given as an install target). Which honestly I don't like either, it's just there as a backcompat measure so that these source releases don't create a regression versus existing sdists -- note that one of the goals of the design was that current sdists could be upgraded into this format by dropping in a single static file (or by an install tool "virtually" dropping in this file when it encounters an old-style sdist -- so you don't need to keep around code to handle both cases separately). Does that sound right? (Other features of the original proposal include stuff like the lack of trivial metadata like "name" and "description", and the support for generating multiple wheels from one directory. I am explicitly calling these "inessential".) -n -- Nathaniel J. Smith -- http://vorpus.org
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig