On 2 June 2017 at 16:27, Donald Stufft <don...@stufft.io> wrote: > So my preference is that everything goes through the sdist step as I think > that is most likely to provide consistent builds everywhere both from a VCS > checkout and from a sdist that was released to PyPI.
Agreed. That's the ideal workflow. The only reason we don't do it now is because... well, I'm not quite sure. I think it's to do with things like setuptools_scm not generating suitable "temporary version numbers" to allow us to work properly with installs that assume that name/version uniquely identifies the code. But regardless, I'd like to say that under PEP 517, we will go source tree -> sdist -> wheel -> install, for everything but editable installs. If projects like setuptools_scm will have an issue with this, they need to feed their requirements into the process of agreeing PEP 517 - the pip developers can't really argue their case for them. > That being said, I am > somewhat sympathetic to the idea that generating a sdist might be a slow > process for reasons that are unrelated to actually building a wheel (for > example, documentation might get “compiled” from some kind of source format > to a man page, html docs, etc) so I think I am not against the idea of > having an optional hook whose job is to just do the copying needed. The > requirements would be: Agreed. Optimising the process is perfectly OK with me, but I do think we should treat it as just that, an optimisation, and require that backends implementing the optimisation route must ensure that it gives the same results as going via a sdist. Note that there's an implication here - if we define the build process in terms of the effect of "going via a sdist", then we need to at least have an intuitive understanding of what that means in practice. I don't think it's a contentious point (even if the specific term "sdist" is open to debate), as I think repeatable builds are a well-understood idea. (It's at this point that the concerns of people who want incremental builds come in - we should support incremental builds in a way that preserves the "just like going via a sdist" principle. But again, they need to raise their concerns if they think we're missing something key to their use case). > * The build_sdist hook is mandatory, but may fail (as any of these commands > may fail tbh) if some invariant required by the build backend isn’t > satisfied. Agreed. Other tools (or future additions to pip), that want to provide a common interface to the "build a sdist" functionality would use this hook too. They may not be able to fall back in the same way as pip can, but that's their issue to address. > * The copy_the_files hook is optional, if it exists it SHOULD produce a tree > that when the build_wheel hook is called in it, will produce a wheel that is > equivalent to one that would have been built had the build_sdist hook been > called instead. This is precisely the "should look like we built a sdist" principle, so I'm a solid +1 on this, too. It might be worth stating that copy_the_files is only intended to be called after a failed call to build_sdist. I don't know if backends would care, but I don't think we should worry about having to support use of copy_the_files as anything other than a build_sdist fallback. > * If the copy_the_files hook is not defined, then the build frontend is free > to just directory call the build_sdist command instead. Sorry? I assume here that you mean "directly call the build_wheel hook in the original source tree"? That's OK, but I think we should be clear that if this happens, it is the backend's responsibility to ensure that the build is equivalent to building from a sdist. It might even be appropriate for the front end to warn if this happens - "Unable to build out of tree - results may differ from a clean build" (The intent is to remind people that they aren't testing the actual sdist they will be deploying, the issue that Nick pointed out). One thing that is worth flagging here, if only as a note for backends considering this approach, is that the source tree could have arbitrary out of date build artifacts in it (from previous build_wheel calls, possibly with different settings, or from the source tree being installed editable, or even from the user doing something like a manual debug build) and the backend must take responsibility for ensuring that those artifacts don't affect the result. (In case it's not obvious, my personal feeling is that this is a pretty risky option, and I'd strongly prefer backends that implement at least one of the hooks allowing out-of-tree builds). > I think that represents a pretty reasonable trade off, the path of least > resistance for a build backend is to just define build_sdist and build_wheel > and leave the two optional hooks omitted. I suspect for a lot of pure python > packages (although Thomas has said not flit) those two hooks will be fast > enough that is all they’ll need to implement. However in cases they’re not > we provide both the copy_the_files and the wheel_metadata hook to allow > short circuiting a possibly more complex build process to provide a better > UX to end users. That kinds of goes against my “good intentions don’t > matter” statement from before, but I also think that practicality beats > purity ;) Agreed, this is looking reasonable to me. I think it covers pip's requirements, and I hope it addresses Thomas' needs for flit. I don't honestly have a feel for what other backends might look like, so I'll leave other people to comment on those. Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
