On Thu, Jun 1, 2017 at 1:22 PM, Donald Stufft <don...@stufft.io> wrote:
> > On Jun 1, 2017, at 2:12 PM, C Anthony Risinger <c...@anthonyrisinger.com> > wrote: > > Because the build itself can output additional source files, that may be > desirable to include in an sdist later, I honestly don't think you can pass > through a "proper" sdist before a wheel. I think you can 99% of the time do > that, but some builds using Cython and friends could actually have a custom > initial build that generates standard .h/.c/.py, and even outputs an > alternative p > yproject.toml that *no longer needs* a custom build backend. Or just > straight deletes it from SOURCE-RECORD once the custom build is done, > because some artifacts are enough to rebuild a wheel next time. It seems to > me the only possibly correct order is: > > 1. VCS checkout > 2. partial sdist, but still likely an sdist, no promises! > 3. wheel > 4. proper sdist from generated SOURCE-RECORD, or updated static > SOURCE-RECORD, or just original source tree + DIST-INFO > > I don't see a way to get a 100% valid sdist without first building the > project and effectively asking the build backend (via its SOURCE-RECORD, if > any) "Well golly, you did a build! What, *from both the source tree and > build artifacts*, is important for wrapping up into a redistributable?" > > Maybe I'm overlooking something yuge (I've tried to follow this > discussion, and have sort of checked out of python lately, but I'm fairly > well-versed in packing lore and code), but in general I think we really are > making sdists way way... way scarier than need be. They're pretty much > whatever the build tells you is important for redistribution, at the end, > with as much static meta > data as possible, to the point of possibly obviating their need for > pyproject.toml in the first place... maybe this aspect is what is hanging > everyone up? A redistibutable source does not need to be as flexible as the > original VCS input. An sdist is pinned to a specific version of a project, > whereas VCS represents all possible versions (albeit only one is checkout > out), and sdists > *are not* wheels! The same expectations need not apply. Two sdists of the > same version might not be identical; one might request the custom build > backed via pyproject.toml, and the other might have already done some of > the steps for whatever reason. Authors must decide which is more > appropriate for sharing. > > > > Do any projects build a copy of the library and use that for influencing > what gets copied into the sdist today? As far as I am aware there is not > any that do that. I think the standard thing to do in Cython is to produce > the .c files as part of the sdist process, which is perfectly fine to do. > With the newer PEPs it doesn’t _need_ to do that, since you can depend on > Cython in your build steps and just ship the pyx files (although you’re > still free to compute the .c files AOT and include them in the sdist). > I admit I don't know of any either. Nor did I know the standard expectation in Cython was to generate things in the sdist phase. I have not personally used it for a project and was only using it as an example of a process that produces potentially redistributable artifacts. What you are saying though I think only emphasizes previous comments about needing to pin down, once and for all, what it means to "be an sdist" (which I agree with), right now, and decide who is responsible for constructing that. We don't need to go on a rampage pronouncing new formats or files requirements like LICENCE... just state the status-quo and accept it. Maybe we can start with simply defining an sdist as "some tree with a DIST-INFO". I'll avoid the package-name.dist-info for now and the problems therein, unless there is a simple consensus there. >From that, there seems to only be a small gap in the build api hooks, and a missing "[sdist-system]" phase (maybe that doesn't sound as nice as build-system) that I believe would be a small PEP or additional to 517. In all honesty, I think probably *both* the sdist-system and the build-system need to be consulted to fully populate DIST-INFO... this can be the `build_sdist` hook in both. In all honestly, as you have clarified, Cython is *not* a build-system! It's an sdist-system. The C compiler is the expected build-system. For many build-systems, `build_sdist` might even be a noop, but it might still want the opportunity to make adjustments before starting. It seems reasonable to me that both systems simply begin from a static DIST-INFO, if any, then work together to populate and update it. Something like setuptools_scm, as a sdist-system, might only generate a barebones METADATA with name and version info, then the selected build-system comes in and fills the rest. Or something like the Cython sdist-system might generate more source files and not even touch DIST-INFO, then the selected build-system comes in and fills the rest. Neither have anything to do with "the build". Something like Cython is effectively doing a partial build before producing a redistributable source tree, and if we skip that step and go straight to a build via the build-system, then the only real option for sdists at that point is to ask the same backend, post-build, for the important redistibutable parts, which may or may not reflect a stable/reproducible input to the same system. If I select "Cython" as the build-system, but I need to use a different compiler for currently-unsupported-platform-X, I'm going to have a bad day. -- C Anthony
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
