On 17 July 2017 at 15:41, Thomas Kluyver <tho...@kluyver.me.uk> wrote: > [I don't have time to properly read & respond now, but I wanted to > suggest something] > > Would it be better if we said that build_directory must always be > specified, i.e. passing None is invalid? > > A) Nathaniel has quite rightly expressed concern over requiring two > different code paths. While I think the extra work is less than his > wording implies, it is an extra case for things to go wrong.
I don't think that makes sense, as we want a clear "you don't need to worry about sdist consistency" code path for the normal case where the frontend has already ensured that the source directory is an unpacked sdist (hence ensuring that the built wheel will be consistent with that sdist as long as the backend restricts itself to using the contents of that directory as inputs). This core requirement seems to have gotten lost for a lot of folks due to the recent focus on how best to allow frontends like pip to handle the "Here's an arbitrary directory, please build it" case, when we expect the typical end user cases to be ones where the frontend already knows it has a pristine unpacked sdist and simply needs the backend to turn that directory into an installable wheel file (either because it download the sdist from PyPI, or because it just created it via build_sdist). While it would be lovely to be able to go with the "let's just pretend that problem doesn't exist for now" approach for the arbitrary directory case, I think you're right that it's a common enough scenario that we can't ignore it entirely, even in the first iteration of the API specification - to enable and encourage adoption, we need to give frontends and backends a reasonable way to exchange information about what is going on, and preferably do so in a way that aligns nicely with design concepts that already exist as part of full-fledged build systems. That last point is why I specifically *don't* want an idiosyncratic Python-specific flag as part of the API, and would prefer instead to address the need as a conventional expectation around the behaviour of out-of-tree builds based on exemplars like flit and enscons, similar to the way the semantics of operator overloading are handled in Python itself (i.e. technically you can do whatever you want with operator overloading, but if your overloading choices are inconsistent with other conventional uses of those symbols and hence cause problems for people and makes your API confusing or unreliable, they may decide not to use your library because of it). In-place/out-of-tree is an inherently meaningful concept for arbitrary build backends, and something that they can't automatically choose for themselves (since they need the target directory as a user provided configuration setting). By contrast, whether the starting directory for build_wheel is an unpacked sdist or not is something a backend should be able to work out for itself (e.g. by checking for PKG-INFO). That said, I *might* accept an explicit "unpacked_sdist=True/False" flag if someone can provide a compelling justification for why it isn't redundant with "build_directory=None" and/or backends checking directly for PKG-INFO (or other files they add to the sdist) in the source directory, and an explanation of what backends should do for "unpacked_sdist=False, build_directory=None" that they wouldn't do for the "unpacked_sdist=True, build_directory=None" case. Unlike build_directory, they certainly couldn't pass such a flag straight on to a general purpose build backend (or not pass it, in the "build_directory=None" case). And if the proposal is instead for an unpacked_sdist flag as a *replacement* for the currently proposed build_directory parameter... well, no. Out-of-tree builds are useful for a lot more than just pip indicating whether or not it failed to build an sdist, and if the claim is "there's no evidence that out-of-tree builds are sufficiently useful to include them in the specification", then I'll point you to the laments of everyone trying to manage cross-compiling for multiple platform ABIs without them, both inside and outside the Python community, as well as the efforts to design sensible ecosystem independent intermediate artifact caching strategies for automated build pipelines. Those reasons aren't necessarily part of why the parameter was initially proposed, but they *are* a big part of why I was an instant convert to the idea once the suggestion was made (and why I've actively opposed the notion that the concept hasn't demonstrated its utility well enough to be incorporated). > B) The current spec does another 'semantics not specified' for in place > builds. This should make us uncomfortable - we're asking backends to > implement something without telling them what. When we did something > similar for editable installs, we ended up pulling it out again. Technically the "semantics unspecified" only applies when the source directory is something other than an unpacked sdist, since such a build may differ from building via the sdist in unspecified, backend dependent ways. (This isn't new, it's acknowledging that the handling of dirty source directories is an inherently backend dependent behaviour, and that the main problem that pip currently has is with the way that setuptools/distutils/setup.py in particular deal with it, *not* necessarily with the way future backends will handle it in general). The semantics specification we add for the out-of-tree case is that it should be as close to "build sdist -> unpack sdist -> in-place build wheel" as practical in the given environment, while still leaving the specifics of how that is implemented, and how close it gets to actually matching the build-via-sdist behaviour up to the backend. For example, you're free to decide as flit's developer that since frontends are free to try the "build_sdist" path for themselves, you *won't* implicitly implement that for out-of-tree builds in flit, and will instead handle out-of-tree builds exactly the same way as you handle in-place builds. Whether or not that is sufficient in practice will then be an implementation discussion between you and your users (and *maybe* frontend developers if they end up fielding a lot of flit-specific bug reports), *not* an API specification discussion at the PyPA/distutils-sig level. For enscons, Daniel may decide to handle out-of-tree builds by passing "build_directory" down to the Scons backend as the target "variant directory", and otherwise not do anything special. Again, an entirely acceptable way of handling the proposed API specification. I'm entirely open to the idea that this part of the PEP still needs clarification - what's there attempts to explain why I'm willing to accept it as BDFL-Delegate, and why the pip developers are willing to accept it as sufficient for their needs, but it's also important that other frontend developers are able to understand how to use it, and backend developers are able to understand the anticipated expectations around how they implement it. > C) We thought we were adding this to satisfy people who want incremental > builds, but it now looks like we were conflating two different things, > and this is not actually a good answer for those use cases. It does give folks a backend independent way to request incremental builds, but the main use case in practice is as the baseline essential build step of going from an unpacked sdist directory to a wheel file. > D) The frontend can specify build_directory=source_dir/build to build in > a subdirectory of the source tree. While this is true, I don't think it needs to be called out explicitly in the PEP. > E) When we do work out the need and the semantics for in place builds, > we can write another PEP adding an optional hook for that. The minimal specification for in-place builds is "Whatever you would do to build a wheel file from an unpacked sdist". The unspecified part is how that behaviour may change in the case where the source directory *isn't* an unpacked sdist and hence may contain other files. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
