Re: [Distutils] A possible refactor/streamlining of PEP 517
> On Jul 15, 2017, at 11:50 PM, Nick Coghlan wrote: > > The exact norms around what's acceptable behaviour for out-of-tree > wheel builds (and just how hard backends should try to match the > build_sdist -> in-place build_wheel path in that case) is then > something that will evolve over time, and I'm OK with that. My expectation is that build backends are going to do the same amount of effort for trying to match the “via sdist” case in both the out of place and the in-place builds. I may be wrong, but that’s my general expectation because I don’t think it really makes sense for a build backend to go through a whole lot of extra effort in the out of place build case and then.. just not do that in the in-place build case. So I think that option is largely satisfying the “don’t crap up the current directory” desire and a desire to put the build artifacts in a certain location for subsequent caching. I don’t think it’s going to be generally useful for trying to match a sdist. I don’t however think this is a bad thing, honestly trying to do something different in terms of matching in-place vs out-of-place builds in terms of what files get installed sounds like a recipe for adding *another* variant of way something gets installed which means it makes the possible problem worse not better. I think that if you want a guarantee of parity with the via sdist case, then you have to go via sdist or you’re just introducing another case, but we should document that build tools should generally strive to be as consistent as they can in both the via sdist and the direct to wheel case. — Donald Stufft ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On Sat, Jul 15, 2017, at 08:33 PM, Donald Stufft wrote: > I wonder if maybe it would be more useful to simply recommend that instead of > shelling out to random vcs binaries that these projects depend on (or bundle) > libraries to directly interact with a repository. For instance, if your > project supports git, then you can use dulwich or pygit2 and then the > invariant of “building inside of a docker container without `git` installed” > still remains functional. I did consider this kind of approach. It might be feasible for git using dulwich (pygit2 expects libgit2 on your system, so you can't just require it as a Python package). But it's ironically not workable with mercurial, even though it's pure Python, because hg uses Python 2, while flit requires Python 3. And I don't see this working reliably for svn, or bzr, or other less common VCSs. So at least for flit, I think we will continue to rely on external, non-pip installable dependencies for this. This isn't a problem so long as building an sdist isn't necessary to get a project installed. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On Sat, Jul 15, 2017 at 11:27 PM, Nick Coghlan wrote: > On 16 July 2017 at 14:56, Nathaniel Smith wrote: >> But... that is not what the in-place/out-of-place distinction means in >> normal usage, it's not the distinction that any of those build systems >> you were surveying implement, and it's not the distinction specified >> in the current PEP text. >> >> If what we want is a distinction between "please give me a correct >> wheel" and "please give me a wheel but I don't care if it's broken", >> then wouldn't it make more sense to have a simple flag saying *that*? > > No, because pip *also* wants the ability to request that the backend > put the intermediate build artifacts in a particular place, Say what? Where did they say this? I'm 99% sure this is just not true. > *and* > having that ability will likely prove beneficial given directory based > caching schemes in build automation pipelines (with BitBucket > Pipelines and OpenShift Image Streams being the two I'm personally > familiar with, but it's a logical enough approach to speeding up build > pipelines that I'm sure there are others). Yeah, this is a really neat idea! I'm genuinely enthusiastic about it. Which... is why I think this is a great target for a future PEP, that can adequately address the complications that the current PEP is skimming over. As I argued in a previous email, I think these pipelines would actually *prefer* that out-of-place build be an optional feature, but it's basically impossible to even have the discussion about what they want properly as part of the core PEP 517 discussion. > It just turns out that we can piggy back off that in-place/out-of-tree > distinction to *also* indicate how much the frontend cares about > consistency with sdist builds (which the PEP previously *didn't* say, > but explicit text along those lines was added as part of > https://github.com/python/peps/pull/310/files based on this latest > discussion). Okay, but then this is... bad. You're taking two unrelated distinctions (in-place/out-of-place and sloppy/precise) and smashing them together. In particular, one of the types of build that Donald has said that he considers "sloppy" and worries about avoiding is any kind of incremental build. So if we take Donald's concern and your new PEP text literally, then it rules out incremental out-of-place builds. But incremental out-of-place builds are exactly what you need for the build pipeline case that you're citing as a motivation for this feature. Your PEP is trying to do too many things at once, and that means it's going to do them poorly. >> And in what case would a frontend ever set this >> give_me_a_correct_wheel flag to False? > > When the frontend either genuinely doesn't care (hopefully rare, but > not inconceivable), or else when its building from an unpacked sdist > and hence can be confident that the artifacts will be consistent with > each other regardless of how the backend handles the situation > (expected to be very common, since it's the path that will be followed > for sdists published to PyPI, and when handed an arbitrary PEP 517 > source tree to build, pip will likely try "build_sdist -> in-place > build_wheel" first and only fall back to "out-of-tree build_wheel" if > the initial build_sdist call fails). Ah, the unpacked sdist case is a good point, I neglected that in my discussion of a possible setuptools build_wheel hook. But it's fine -- even if we say that the setuptools build_wheel hook has to produce an "sdist-consistent" wheel in all cases, then it can detect whether it's building from an unpacked sdist (e.g. by keying off the presence of PKG-INFO), and in that case it knows MANIFEST.in has already been taken into account and it can go straight to bdist_wheel. And in fact, this is what we *want* it to key off of, *not* the in-place/out-of-place thing. Consider Debian: they want to do out-of-place builds of unpacked sdists. They're working with pristine unpacked sdists, so they don't want to setuptools to go pack up a new sdist and then unpack it again for every out-of-place build. (In fact, this is probably a less-tested path than building directly from the unpacked sdist.) So if the point of the out-of-place build feature is that it's supposed to tell setuptools that it needs to go via sdist... then in this case it will do the wrong thing. Out-of-place and sdist-consistency are orthogonal concepts. > This is the main reason piggy backing off the in-place/out-of-tree > distinction works so well for this purpose: if the frontend just > unpacked an sdist into a build directory, then the most obvious thing > for it to do is to do an in-place build in that directory. > > It's only when the frontend is handed an arbitrary directory to build > that it doesn't know for sure is an unpacked sdist that the right > thing to do becomes markedly less clear, which is why we're offering > three options: > > 1. build_sdist -> unpack sdist -> in-place build_wheel (same as a PyPI > do
Re: [Distutils] A possible refactor/streamlining of PEP 517
Just throwing in a vote, since I am following along – Nathaniel is totally correct and I have no idea what Nick is talking about :) In/out-of-place builds can’t guarantee that any “build wheel” operation will be consistent with the “build sdist” operation, except for dealing with a very narrow set of bugs. If you want wheels and sdists to be the same, require it of backends through the specification – not the API. About the only option available here in the protocol would be a “strict” flag, which might allow backends to fail immediately if they can’t guarantee it (e.g. the example backend), but there is literally no API design that can implicitly force the correct behavior. “Build wheel and make a mess in my repo” vs “build wheel without leaving a mess” (in-tree vs. out of tree) is a useful option, but not a solution to backends producing incorrect output. (In case it’s not clear, I’m using Nathaniel’s definitions, since I don’t understand any of the other definitions people have used.) Cheers, Steve Top-posted from my Windows phone From: Nathaniel Smith Sent: Sunday, July 16, 2017 10:25 To: Nick Coghlan Cc: distutils-sig Subject: Re: [Distutils] A possible refactor/streamlining of PEP 517 On Sat, Jul 15, 2017 at 11:27 PM, Nick Coghlan wrote: > On 16 July 2017 at 14:56, Nathaniel Smith wrote: >> But... that is not what the in-place/out-of-place distinction means in >> normal usage, it's not the distinction that any of those build systems >> you were surveying implement, and it's not the distinction specified >> in the current PEP text. >> >> If what we want is a distinction between "please give me a correct >> wheel" and "please give me a wheel but I don't care if it's broken", >> then wouldn't it make more sense to have a simple flag saying *that*? > > No, because pip *also* wants the ability to request that the backend > put the intermediate build artifacts in a particular place, Say what? Where did they say this? I'm 99% sure this is just not true. > *and* > having that ability will likely prove beneficial given directory based > caching schemes in build automation pipelines (with BitBucket > Pipelines and OpenShift Image Streams being the two I'm personally > familiar with, but it's a logical enough approach to speeding up build > pipelines that I'm sure there are others). Yeah, this is a really neat idea! I'm genuinely enthusiastic about it. Which... is why I think this is a great target for a future PEP, that can adequately address the complications that the current PEP is skimming over. As I argued in a previous email, I think these pipelines would actually *prefer* that out-of-place build be an optional feature, but it's basically impossible to even have the discussion about what they want properly as part of the core PEP 517 discussion. > It just turns out that we can piggy back off that in-place/out-of-tree > distinction to *also* indicate how much the frontend cares about > consistency with sdist builds (which the PEP previously *didn't* say, > but explicit text along those lines was added as part of > https://github.com/python/peps/pull/310/files based on this latest > discussion). Okay, but then this is... bad. You're taking two unrelated distinctions (in-place/out-of-place and sloppy/precise) and smashing them together. In particular, one of the types of build that Donald has said that he considers "sloppy" and worries about avoiding is any kind of incremental build. So if we take Donald's concern and your new PEP text literally, then it rules out incremental out-of-place builds. But incremental out-of-place builds are exactly what you need for the build pipeline case that you're citing as a motivation for this feature. Your PEP is trying to do too many things at once, and that means it's going to do them poorly. >> And in what case would a frontend ever set this >> give_me_a_correct_wheel flag to False? > > When the frontend either genuinely doesn't care (hopefully rare, but > not inconceivable), or else when its building from an unpacked sdist > and hence can be confident that the artifacts will be consistent with > each other regardless of how the backend handles the situation > (expected to be very common, since it's the path that will be followed > for sdists published to PyPI, and when handed an arbitrary PEP 517 > source tree to build, pip will likely try "build_sdist -> in-place > build_wheel" first and only fall back to "out-of-tree build_wheel" if > the initial build_sdist call fails). Ah, the unpacked sdist case is a good point, I neglected that in my discussion of a possible setuptools build_wheel hook. But it's fine -- even if we say that the setuptools build_wheel hook has to produce an "sdist-consistent" wheel in all cases, then it can detect whether it's building from an unpacked sdist (e.g. by keying off the presence of PKG-INFO), and in that case it knows MANIFEST.in has already been taken into account and it can go straight to bdist_wheel. And
Re: [Distutils] A possible refactor/streamlining of PEP 517
I agree that sdist consistency is not enforceable. Very little is. What if we deleted every unenforceable part of the PEP? No explanations of what backends should do. Every parameter is a hint. If you put the output file where requested then you are a good back end. Would that work better? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
That totally works for me. It also avoids the argument we seem to be having around tricking backends into creating the correct output by controlling their input. It’s the backend job to get the output correct – pip can output as many messages as it likes to blame someone else, if the aim is to avoid bugs being reported in the wrong place, but at some point we need to trust the backend to get it right even in a dirty source directory. Top-posted from my Windows phone at EuroPython From: Daniel Holth Sent: Sunday, July 16, 2017 21:18 To: Steve Dower; Nathaniel Smith; Nick Coghlan Cc: distutils-sig Subject: Re: [Distutils] A possible refactor/streamlining of PEP 517 I agree that sdist consistency is not enforceable. Very little is. What if we deleted every unenforceable part of the PEP? No explanations of what backends should do. Every parameter is a hint. If you put the output file where requested then you are a good back end. Would that work better? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 16 July 2017 at 18:24, Nathaniel Smith wrote: > On Sat, Jul 15, 2017 at 11:27 PM, Nick Coghlan wrote: >> On 16 July 2017 at 14:56, Nathaniel Smith wrote: >>> But... that is not what the in-place/out-of-place distinction means in >>> normal usage, it's not the distinction that any of those build systems >>> you were surveying implement, and it's not the distinction specified >>> in the current PEP text. >>> >>> If what we want is a distinction between "please give me a correct >>> wheel" and "please give me a wheel but I don't care if it's broken", >>> then wouldn't it make more sense to have a simple flag saying *that*? >> >> No, because pip *also* wants the ability to request that the backend >> put the intermediate build artifacts in a particular place, > > Say what? Where did they say this? I'm 99% sure this is just not true. pip currently works by moving the input files out to a different directory with shutil.copytree. For PEP 517, their default build strategy is going to be to take an sdist and unpack it. Both of those build strategies are particular ways of achieving an out-of-tree build *without* relying directly on out-of-tree build support in the backend. However, if build_sdist fails, they *can't* use their preferred build strategy, and need a way to ask the backend to do the best it reasonably can to emulate a build via sdist, even though an sdist build isn't technically feasible. Originally, the proposed mechanism for that was the various incarnations of the "prepare_input_for_build_wheel" hook, which had the key downside of not aligning well with the way full-fledged build systems actually work. By contrast, Daniel & Thomas's suggestion of a build_directory parameter to build wheel nicely models a front end saying "Well, I *would* have built an sdist and used that, but it didn't work, so instead I'm asking you to do the best you can" in a way that *also* aligns nicely with the way out-of-tree build support in full-fledged build systems actually works (i.e. by specifying target directories/build directories/variant directories appropriately) >> *and* >> having that ability will likely prove beneficial given directory based >> caching schemes in build automation pipelines (with BitBucket >> Pipelines and OpenShift Image Streams being the two I'm personally >> familiar with, but it's a logical enough approach to speeding up build >> pipelines that I'm sure there are others). > > Yeah, this is a really neat idea! I'm genuinely enthusiastic about it. > Which... is why I think this is a great target for a future PEP, that > can adequately address the complications that the current PEP is > skimming over. As I argued in a previous email, I think these > pipelines would actually *prefer* that out-of-place build be an > optional feature, but it's basically impossible to even have the > discussion about what they want properly as part of the core PEP 517 > discussion. Even with build_directory defined, backends remain free to ignore it and put their intermediate artifacts somewhere else. All the API design does is provide them with the *option* of using the frontend provided directory. So that just becomes a quality of implementation issue that folks will work out iteratively with backend developers. >> It just turns out that we can piggy back off that in-place/out-of-tree >> distinction to *also* indicate how much the frontend cares about >> consistency with sdist builds (which the PEP previously *didn't* say, >> but explicit text along those lines was added as part of >> https://github.com/python/peps/pull/310/files based on this latest >> discussion). > > Okay, but then this is... bad. You're taking two unrelated > distinctions (in-place/out-of-place and sloppy/precise) and smashing > them together. In particular, one of the types of build that Donald > has said that he considers "sloppy" and worries about avoiding is any > kind of incremental build. So if we take Donald's concern and your new > PEP text literally, then it rules out incremental out-of-place builds. > But incremental out-of-place builds are exactly what you need for the > build pipeline case that you're citing as a motivation for this > feature. > > Your PEP is trying to do too many things at once, and that means it's > going to do them poorly. Not really, because sloppy/precise isn't actually a distinction we really care about, it's one that arises from the fact that in-place builds with setuptools/distutils *are* sloppy about their inputs, and hence starting with a "dirty" source directory may corrupt the sdists and wheels produced from those directories. Full-fledged build systems that work backwards through their dependency graph to their desired inputs rather than relying on file globs and collecting entire directory trees from local disk shouldn't be anywhere near as vulnerable to the problem (although even for those the general recommendation is that robust CI and release processes should always start with a c
Re: [Distutils] A possible refactor/streamlining of PEP 517
Nathaniel Smith wrote: I think at the moment it's basically only you and me who actually have strong opinions, and everyone else is barely following. I've been a somewhat confused bystander in this thread for a while, and for what it's worth, I think Nathaniel is exactly right on these points: 1. In-place vs. out-of place and sdist/wheel consistency are orthogonal issues. 2. Sdist/wheel consistency should be an implied goal that goes without saying in all cases. Even if a backend can't always guarantee it, it should make a best effort. -- Greg ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
[I don't have time to properly read & respond now, but I wanted to suggest something] Would it be better if we said that build_directory must always be specified, i.e. passing None is invalid? A) Nathaniel has quite rightly expressed concern over requiring two different code paths. While I think the extra work is less than his wording implies, it is an extra case for things to go wrong. B) The current spec does another 'semantics not specified' for in place builds. This should make us uncomfortable - we're asking backends to implement something without telling them what. When we did something similar for editable installs, we ended up pulling it out again. C) We thought we were adding this to satisfy people who want incremental builds, but it now looks like we were conflating two different things, and this is not actually a good answer for those use cases. D) The frontend can specify build_directory=source_dir/build to build in a subdirectory of the source tree. E) When we do work out the need and the semantics for in place builds, we can write another PEP adding an optional hook for that. Thomas On Mon, Jul 17, 2017, at 04:44 AM, Nick Coghlan wrote: > On 16 July 2017 at 18:24, Nathaniel Smith wrote: > > On Sat, Jul 15, 2017 at 11:27 PM, Nick Coghlan wrote: > >> On 16 July 2017 at 14:56, Nathaniel Smith wrote: > >>> But... that is not what the in-place/out-of-place distinction means in > >>> normal usage, it's not the distinction that any of those build systems > >>> you were surveying implement, and it's not the distinction specified > >>> in the current PEP text. > >>> > >>> If what we want is a distinction between "please give me a correct > >>> wheel" and "please give me a wheel but I don't care if it's broken", > >>> then wouldn't it make more sense to have a simple flag saying *that*? > >> > >> No, because pip *also* wants the ability to request that the backend > >> put the intermediate build artifacts in a particular place, > > > > Say what? Where did they say this? I'm 99% sure this is just not true. > > pip currently works by moving the input files out to a different > directory with shutil.copytree. > > For PEP 517, their default build strategy is going to be to take an > sdist and unpack it. > > Both of those build strategies are particular ways of achieving an > out-of-tree build *without* relying directly on out-of-tree build > support in the backend. > > However, if build_sdist fails, they *can't* use their preferred build > strategy, and need a way to ask the backend to do the best it > reasonably can to emulate a build via sdist, even though an sdist > build isn't technically feasible. > > Originally, the proposed mechanism for that was the various > incarnations of the "prepare_input_for_build_wheel" hook, which had > the key downside of not aligning well with the way full-fledged build > systems actually work. > > By contrast, Daniel & Thomas's suggestion of a build_directory > parameter to build wheel nicely models a front end saying "Well, I > *would* have built an sdist and used that, but it didn't work, so > instead I'm asking you to do the best you can" in a way that *also* > aligns nicely with the way out-of-tree build support in full-fledged > build systems actually works (i.e. by specifying target > directories/build directories/variant directories appropriately) > > >> *and* > >> having that ability will likely prove beneficial given directory based > >> caching schemes in build automation pipelines (with BitBucket > >> Pipelines and OpenShift Image Streams being the two I'm personally > >> familiar with, but it's a logical enough approach to speeding up build > >> pipelines that I'm sure there are others). > > > > Yeah, this is a really neat idea! I'm genuinely enthusiastic about it. > > Which... is why I think this is a great target for a future PEP, that > > can adequately address the complications that the current PEP is > > skimming over. As I argued in a previous email, I think these > > pipelines would actually *prefer* that out-of-place build be an > > optional feature, but it's basically impossible to even have the > > discussion about what they want properly as part of the core PEP 517 > > discussion. > > Even with build_directory defined, backends remain free to ignore it > and put their intermediate artifacts somewhere else. All the API > design does is provide them with the *option* of using the frontend > provided directory. > > So that just becomes a quality of implementation issue that folks will > work out iteratively with backend developers. > > >> It just turns out that we can piggy back off that in-place/out-of-tree > >> distinction to *also* indicate how much the frontend cares about > >> consistency with sdist builds (which the PEP previously *didn't* say, > >> but explicit text along those lines was added as part of > >> https://github.com/python/peps/pull/310/files based on this latest > >> discussion). > > > > Okay, but then this is... b
