Re: [Distutils] A possible refactor/streamlining of PEP 517
On Mon, Jul 10, 2017, at 07:01 AM, Nick Coghlan wrote: > So I think we have pretty solid evidence that the reason the > procedural "build directory preparation" hook wasn't sitting well with > people was because that isn't the way build systems typically model > the concept, while a "build directory" setting is very common (even if > that "setting" is "the current working directory when configuring or > running the build"). Hooray! :-) Do we want to also provide a build_directory for the build_sdist hook? In principle, I don't think making an sdist should involve a build step, but I know that some projects do perform steps like cython code gen or JS minification before making the sdist. I think this was a workaround to ease installation before wheel support was widespread, and I'd be inclined to discourage it now, so my preference would be no build_directory parameter for build_sdist. Backends which insist on generating intermediates at that point can make a temp dir themselves. Then I guess that the choice between building a wheel directly and attempting to build an sdist first (with direct fallback) is one for frontends, and doesn't need to be specified. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
One nice thing about providing a “put your work in this directory” setting for all tasks is that only the front end has to know how and where to create it, and how and when to clean it up later. Users may want to configure this across all projects, regardless of the backend in use. Permitting this directory to be the source tree implicitly requires backends to support “in place” builds (i.e. you should put output files in a matching structure under that directory in case it really is the source tree). In this case, front ends need to be responsible for (not) running rmtree and backends should not blindly delete everything (or else they’ll get bug reports from very upset users). Cheers, Steve Top-posted from my Windows phone at EuroPython From: Thomas Kluyver Sent: Monday, July 10, 2017 9:14 To: distutils-sig@python.org Subject: Re: [Distutils] A possible refactor/streamlining of PEP 517 On Mon, Jul 10, 2017, at 07:01 AM, Nick Coghlan wrote: > So I think we have pretty solid evidence that the reason the > procedural "build directory preparation" hook wasn't sitting well with > people was because that isn't the way build systems typically model > the concept, while a "build directory" setting is very common (even if > that "setting" is "the current working directory when configuring or > running the build"). Hooray! :-) Do we want to also provide a build_directory for the build_sdist hook? In principle, I don't think making an sdist should involve a build step, but I know that some projects do perform steps like cython code gen or JS minification before making the sdist. I think this was a workaround to ease installation before wheel support was widespread, and I'd be inclined to discourage it now, so my preference would be no build_directory parameter for build_sdist. Backends which insist on generating intermediates at that point can make a temp dir themselves. Then I guess that the choice between building a wheel directly and attempting to build an sdist first (with direct fallback) is one for frontends, and doesn't need to be specified. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On Mon, Jul 10, 2017 at 7:13 PM, Thomas Kluyver wrote: > On Mon, Jul 10, 2017, at 07:01 AM, Nick Coghlan wrote: > > So I think we have pretty solid evidence that the reason the > > procedural "build directory preparation" hook wasn't sitting well with > > people was because that isn't the way build systems typically model > > the concept, while a "build directory" setting is very common (even if > > that "setting" is "the current working directory when configuring or > > running the build"). > > Hooray! :-) > > Do we want to also provide a build_directory for the build_sdist hook? > In principle, I don't think making an sdist should involve a build step, > but I know that some projects do perform steps like cython code gen or > JS minification before making the sdist. I think this was a workaround > to ease installation before wheel support was widespread, and I'd be > inclined to discourage it now, so my preference would be no > build_directory parameter for build_sdist. Backends which insist on > generating intermediates at that point can make a temp dir themselves. > No preference on yes/no for build_directory for build_sdist hook, but invoking Cython on .pyx files to generate C code rather than checking in generated C code is good practice. I don't think we want to go back to checking in generated code, nor do we want to store generated code in tmpdirs (because that loses the advantage of not having to regenerate when .pyx hashes are identical). Ralf > Then I guess that the choice between building a wheel directly and > attempting to build an sdist first (with direct fallback) is one for > frontends, and doesn't need to be specified. > > Thomas > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 10 July 2017 at 17:13, Thomas Kluyver wrote: > Do we want to also provide a build_directory for the build_sdist hook? > In principle, I don't think making an sdist should involve a build step, > but I know that some projects do perform steps like cython code gen or > JS minification before making the sdist. I think this was a workaround > to ease installation before wheel support was widespread, and I'd be > inclined to discourage it now, so my preference would be no > build_directory parameter for build_sdist. Backends which insist on > generating intermediates at that point can make a temp dir themselves. Agreed, I think the two kinds of artifact generation we want to support directly in PEP 517 are: - artifacts checked in to source control (ala autotools and CPython's Argument Clinic) - artifacts created while building the wheel file While we can't *prevent* backends doing artifact generation in build_sdist, we don't really want to encourage it either - as you say, we'd prefer that folks migrate such processes to the sdist->wheel step. > Then I guess that the choice between building a wheel directly and > attempting to build an sdist first (with direct fallback) is one for > frontends, and doesn't need to be specified. Yep, and frontends may even decide to do something like delegating out-of-tree build support to backends by default, while offering an option to force the use of an intermediate sdist, similar to the way that tox uses an sdist by default, but provides a setting in tox.ini to turn that off. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 10 July 2017 at 18:56, Ralf Gommers wrote: > On Mon, Jul 10, 2017 at 7:13 PM, Thomas Kluyver > wrote: >> >> On Mon, Jul 10, 2017, at 07:01 AM, Nick Coghlan wrote: >> > So I think we have pretty solid evidence that the reason the >> > procedural "build directory preparation" hook wasn't sitting well with >> > people was because that isn't the way build systems typically model >> > the concept, while a "build directory" setting is very common (even if >> > that "setting" is "the current working directory when configuring or >> > running the build"). >> >> Hooray! :-) >> >> Do we want to also provide a build_directory for the build_sdist hook? >> In principle, I don't think making an sdist should involve a build step, >> but I know that some projects do perform steps like cython code gen or >> JS minification before making the sdist. I think this was a workaround >> to ease installation before wheel support was widespread, and I'd be >> inclined to discourage it now, so my preference would be no >> build_directory parameter for build_sdist. Backends which insist on >> generating intermediates at that point can make a temp dir themselves. > > No preference on yes/no for build_directory for build_sdist hook, but > invoking Cython on .pyx files to generate C code rather than checking in > generated C code is good practice. I don't think we want to go back to > checking in generated code, nor do we want to store generated code in > tmpdirs (because that loses the advantage of not having to regenerate when > .pyx hashes are identical). If the frontend offers a way to specify a particular build directory, then that kind of caching wouldn't be a problem when regenerating files as part of build_wheel - all the build files would be preserved between runs, not only those that made it into the final wheel file. That's a slightly different question than the one Thomas was asking, which is whether or not we want to support out-of-tree builds for sdist creation, and I don't believe we do, since we don't really *want* folks to be adding generated files to their sdist that they aren't keeping under source control - we'd prefer that such activities were postponed to "build_wheel" now that we have separate source and precompiled distribution formats. If publishers still prefer to go down that path, then they can either use mechanisms like .gitignore to cope with the consequences of doing it in-place, or else use ccache-style caching mechanisms that aren't dependent on exactly where you do the build, and instead depend on the hash of the input artifact (e.g. I'd be astonished if cython couldn't implement something like ccache natively for pyx file compilation). Either way, it won't impact of the experience of folks using pre-compiled wheel files, and for at least some use cases for sdists (e.g. Linux distributions), it's typically preferable to regenerate that kind of file anyway. Cheers, NIck. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On Mon, Jul 10, 2017, 04:54 Steve Dower wrote: > One nice thing about providing a “put your work in this directory” setting > for all tasks is that only the front end has to know how and where to > create it, and how and when to clean it up later. Users may want to > configure this across all projects, regardless of the backend in use. > > > > Permitting this directory to be the source tree implicitly requires > backends to support “in place” builds (i.e. you should put output files in > a matching structure under that directory in case it really is the source > tree). In this case, front ends need to be responsible for (not) running > rmtree and backends should not blindly delete everything (or else they’ll > get bug reports from very upset users). > Not sure about this particular idea. It might have to be a special case and implementation in the backend driver, usually build in a directory and copy file into the source tree. > Cheers, > > Steve > > > > Top-posted from my Windows phone at EuroPython > > > > *From: *Thomas Kluyver > *Sent: *Monday, July 10, 2017 9:14 > *To: *distutils-sig@python.org > > > *Subject: *Re: [Distutils] A possible refactor/streamlining of PEP 517 > > > > On Mon, Jul 10, 2017, at 07:01 AM, Nick Coghlan wrote: > > > So I think we have pretty solid evidence that the reason the > > > procedural "build directory preparation" hook wasn't sitting well with > > > people was because that isn't the way build systems typically model > > > the concept, while a "build directory" setting is very common (even if > > > that "setting" is "the current working directory when configuring or > > > running the build"). > > > > Hooray! :-) > > > > Do we want to also provide a build_directory for the build_sdist hook? > > In principle, I don't think making an sdist should involve a build step, > > but I know that some projects do perform steps like cython code gen or > > JS minification before making the sdist. I think this was a workaround > > to ease installation before wheel support was widespread, and I'd be > > inclined to discourage it now, so my preference would be no > > build_directory parameter for build_sdist. Backends which insist on > > generating intermediates at that point can make a temp dir themselves. > > > > Then I guess that the choice between building a wheel directly and > > attempting to build an sdist first (with direct fallback) is one for > > frontends, and doesn't need to be specified. > > > > Thomas > > ___ > > Distutils-SIG maillist - Distutils-SIG@python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 2017-07-10 20:33:16 +1000 (+1000), Nick Coghlan wrote: [...] > we don't really *want* folks to be adding generated files to their > sdist that they aren't keeping under source control - we'd prefer > that such activities were postponed to "build_wheel" now that we > have separate source and precompiled distribution formats. [...] This is a mildly naive view. The software I'm familiar with is actually attempting to reflect metadata _from_ the source revision control _into_ the sdist because while it's "tracked" there it's not tracked as normal files (version information from tags, change logs from the commit history, contributor lists from commit authorship). The metadata in question is lost by just blindly tarring up tracked files, so needs some mechanism to export and inject as untracked files (from the source revision control perspective) for inclusion in the sdist. -- Jeremy Stanley ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
Sdists contain generated PKG-INFO and .egg-info. I'd prefer to let the build backend manage any sdist build directory. It doesn't provide the same benefit to pip as the configurable wheel build directory. On Mon, Jul 10, 2017, 08:09 Jeremy Stanley wrote: > On 2017-07-10 20:33:16 +1000 (+1000), Nick Coghlan wrote: > [...] > > we don't really *want* folks to be adding generated files to their > > sdist that they aren't keeping under source control - we'd prefer > > that such activities were postponed to "build_wheel" now that we > > have separate source and precompiled distribution formats. > [...] > > This is a mildly naive view. The software I'm familiar with is > actually attempting to reflect metadata _from_ the source revision > control _into_ the sdist because while it's "tracked" there it's not > tracked as normal files (version information from tags, change logs > from the commit history, contributor lists from commit authorship). > The metadata in question is lost by just blindly tarring up tracked > files, so needs some mechanism to export and inject as untracked > files (from the source revision control perspective) for inclusion > in the sdist. > -- > Jeremy Stanley > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 10 July 2017 at 22:08, Jeremy Stanley wrote: > On 2017-07-10 20:33:16 +1000 (+1000), Nick Coghlan wrote: > [...] >> we don't really *want* folks to be adding generated files to their >> sdist that they aren't keeping under source control - we'd prefer >> that such activities were postponed to "build_wheel" now that we >> have separate source and precompiled distribution formats. > [...] > > This is a mildly naive view. The software I'm familiar with is > actually attempting to reflect metadata _from_ the source revision > control _into_ the sdist because while it's "tracked" there it's not > tracked as normal files (version information from tags, change logs > from the commit history, contributor lists from commit authorship). > The metadata in question is lost by just blindly tarring up tracked > files, so needs some mechanism to export and inject as untracked > files (from the source revision control perspective) for inclusion > in the sdist. Right, we know there will be *some* generated files (as Daniel notes, there will typically at least be the metadata files in a setuptools/distutils generated sdist), as well as potentially modifications to some files based on the version history. That isn't the question though - the question is whether we want to actively support folks moving "compilation" like activities (minification, pyx->C conversion, etc) to the sdist generation stage by adding the optional "build_directory" option to "build_sdist" as well. And that's the part where we decided the answer is "No", we only want to support the following configurations: build_sdist: working directory -> target directory build_wheel: working directory -> target directory working directory -> build directory -> target directory In all cases the frontend provides a target directory that is distinct from the current working directory, so backends have a place to put both generated intermediate artifacts *and* the final assembled archive file. The difference is that in the build_wheel case, the frontend can explicitly say "don't put intermediate artifacts in the working directory *or* the target directory, put them in the build directory". Backends are obviously still free to create their own temporary and caching directories that the frontend doesn't know anything about, but that's up to the backend to worry about. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 10 July 2017 at 14:58, Nick Coghlan wrote: > That isn't the question though - the question is whether we want to > actively support folks moving "compilation" like activities > (minification, pyx->C conversion, etc) to the sdist generation stage > by adding the optional "build_directory" option to "build_sdist" as > well. > > And that's the part where we decided the answer is "No", we only want > to support the following configurations: I'm not sure I follow this comment (or if I do, I don't agree with it :-)). I would expect projects that use Cython to have the step to build the C files from the Cython sources as part of the "build sdist" step, so that the sdist contains standard C sources, and end users can build from sdist with a C compiler installed, but without needing to install Cython. That's the standard approach these days, and I'd hope it will be supported under PEP 517. Paul PS Completely off-topic, but since when has gmail's web interface stopped allowing you to highlight a section of a mail and hit "Reply" to get just that section quoted? It seems to have changed very recently, or is it some setting change I might have accidentally made? It's intensely annoying, as it makes it much more time consuming to avoid top-posting :-( (and please, no suggestions that I use an alternative client, my usage pattern makes that impractical). ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
A huge benefit to using non distutils build systems is making it easy to generate files at any step. I just don't think it's worth it to force a particular build directory at the generate sdist phase. Foolish consistentcy. On Mon, Jul 10, 2017, 10:20 Paul Moore wrote: > On 10 July 2017 at 14:58, Nick Coghlan wrote: > > That isn't the question though - the question is whether we want to > > actively support folks moving "compilation" like activities > > (minification, pyx->C conversion, etc) to the sdist generation stage > > by adding the optional "build_directory" option to "build_sdist" as > > well. > > > > And that's the part where we decided the answer is "No", we only want > > to support the following configurations: > > I'm not sure I follow this comment (or if I do, I don't agree with it > :-)). I would expect projects that use Cython to have the step to > build the C files from the Cython sources as part of the "build sdist" > step, so that the sdist contains standard C sources, and end users can > build from sdist with a C compiler installed, but without needing to > install Cython. That's the standard approach these days, and I'd hope > it will be supported under PEP 517. > > Paul > > PS Completely off-topic, but since when has gmail's web interface > stopped allowing you to highlight a section of a mail and hit "Reply" > to get just that section quoted? It seems to have changed very > recently, or is it some setting change I might have accidentally made? > It's intensely annoying, as it makes it much more time consuming to > avoid top-posting :-( (and please, no suggestions that I use an > alternative client, my usage pattern makes that impractical). > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 11 July 2017 at 00:20, Paul Moore wrote: > On 10 July 2017 at 14:58, Nick Coghlan wrote: >> That isn't the question though - the question is whether we want to >> actively support folks moving "compilation" like activities >> (minification, pyx->C conversion, etc) to the sdist generation stage >> by adding the optional "build_directory" option to "build_sdist" as >> well. >> >> And that's the part where we decided the answer is "No", we only want >> to support the following configurations: > > I'm not sure I follow this comment (or if I do, I don't agree with it > :-)). I would expect projects that use Cython to have the step to > build the C files from the Cython sources as part of the "build sdist" > step, so that the sdist contains standard C sources, and end users can > build from sdist with a C compiler installed, but without needing to > install Cython. That's the standard approach these days, and I'd hope > it will be supported under PEP 517. It's supported - what's not being explicitly supported is generating such Cython files in a directory *other than* either the original source directory, or the target directory where the final sdist will also be created. (However, I'd also hope to see that practice start declining in popularity as PEP 517 makes it straightforward to ensure that cython is installed automatically when needed to build from source) Wheel builds are different, since they're expected to generate intermediate artifacts (e.g. object files, debugging symbols, Cython output files not included in the sdist), that *aren't* going to be included in the final wheel file, but may still be useful to keep around to speed up subsequent builds. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On Jul 10, 2017 8:59 AM, "Nick Coghlan" wrote: On 10 July 2017 at 22:08, Jeremy Stanley wrote: > On 2017-07-10 20:33:16 +1000 (+1000), Nick Coghlan wrote: > [...] >> we don't really *want* folks to be adding generated files to their >> sdist that they aren't keeping under source control - we'd prefer >> that such activities were postponed to "build_wheel" now that we >> have separate source and precompiled distribution formats. > [...] > > This is a mildly naive view. The software I'm familiar with is > actually attempting to reflect metadata _from_ the source revision > control _into_ the sdist because while it's "tracked" there it's not > tracked as normal files (version information from tags, change logs > from the commit history, contributor lists from commit authorship). > The metadata in question is lost by just blindly tarring up tracked > files, so needs some mechanism to export and inject as untracked > files (from the source revision control perspective) for inclusion > in the sdist. Right, we know there will be *some* generated files (as Daniel notes, there will typically at least be the metadata files in a setuptools/distutils generated sdist), as well as potentially modifications to some files based on the version history. That isn't the question though - the question is whether we want to actively support folks moving "compilation" like activities (minification, pyx->C conversion, etc) to the sdist generation stage by adding the optional "build_directory" option to "build_sdist" as well. And that's the part where we decided the answer is "No", we only want to support the following configurations: build_sdist: working directory -> target directory build_wheel: working directory -> target directory working directory -> build directory -> target directory In all cases the frontend provides a target directory that is distinct from the current working directory, so backends have a place to put both generated intermediate artifacts *and* the final assembled archive file. When preparing a redistributable archive, we don't want people to first generate difficult or inconvenient artifacts? I always thought that was a major feature of an archive, to reduce the content down to common denominators for verification, reproducibility, and build simplicity, at the expense of not being fully representative of the original build capabilities and likely an irreversible step. I know there's been a lot of discussion here, and I probably missed it, but early on we talked about things like Cython and cffi being part of sdist generation (in setuptools at least). Are these things now expected to be deferred to the wheel building stage, thus adding deps to the final build server, or are we just saying there is less utility in supporting an explicit "artifacts directory" during sdist generation? If the latter (and I think this is the case), that does seem reasonable, I just wanted to confirm that such "compliations" are still expected, or at least permissible, in downstream redistributables (sdists). -- C Anthony ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 11 July 2017 at 00:58, C Anthony Risinger wrote: > When preparing a redistributable archive, we don't want people to first > generate difficult or inconvenient artifacts? I always thought that was a > major feature of an archive, to reduce the content down to common > denominators for verification, reproducibility, and build simplicity, at the > expense of not being fully representative of the original build capabilities > and likely an irreversible step. My apologies folks, this is an entirely irrelevant tangent brought on by my attempting to explain my own preference that source archives (including sdists) actually *be* source archives, containing solely the original software is *its preferred form for modification*. However, that's a free software, and commercial redistributor centric point of view, where we aren't particularly keen on anyone publishing open source software without also properly declaring all the things that redistributors and end users will need in order to actually modify that software. It's neither enforced nor required by PEP 517, it's just part of my *own* rationale for wanting to see "build directory" kept purely as a wheel building concept, rather than something we define for sdists as well. If some projects *do* decide to put a lot of generated platform independent artifacts in their sdists, those of us that care always have the option of bypassing them and going straight to raw VCS clones and tarballs, just as we already ignore wheel files and retrieve sdists from PyPI instead. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On Mon, Jul 10, 2017, 10:58 C Anthony Risinger wrote: > On Jul 10, 2017 8:59 AM, "Nick Coghlan" wrote: > > On 10 July 2017 at 22:08, Jeremy Stanley wrote: > > On 2017-07-10 20:33:16 +1000 (+1000), Nick Coghlan wrote: > > [...] > >> we don't really *want* folks to be adding generated files to their > >> sdist that they aren't keeping under source control - we'd prefer > >> that such activities were postponed to "build_wheel" now that we > >> have separate source and precompiled distribution formats. > > [...] > > > > This is a mildly naive view. The software I'm familiar with is > > actually attempting to reflect metadata _from_ the source revision > > control _into_ the sdist because while it's "tracked" there it's not > > tracked as normal files (version information from tags, change logs > > from the commit history, contributor lists from commit authorship). > > The metadata in question is lost by just blindly tarring up tracked > > files, so needs some mechanism to export and inject as untracked > > files (from the source revision control perspective) for inclusion > > in the sdist. > > Right, we know there will be *some* generated files (as Daniel notes, > there will typically at least be the metadata files in a > setuptools/distutils generated sdist), as well as potentially > modifications to some files based on the version history. > > That isn't the question though - the question is whether we want to > actively support folks moving "compilation" like activities > (minification, pyx->C conversion, etc) to the sdist generation stage > by adding the optional "build_directory" option to "build_sdist" as > well. > > And that's the part where we decided the answer is "No", we only want > to support the following configurations: > > build_sdist: > working directory -> target directory > > build_wheel: > working directory -> target directory > working directory -> build directory -> target directory > > In all cases the frontend provides a target directory that is distinct > from the current working directory, so backends have a place to put > both generated intermediate artifacts *and* the final assembled > archive file. > > > When preparing a redistributable archive, we don't want people to first > generate difficult or inconvenient artifacts? I always thought that was a > major feature of an archive, to reduce the content down to common > denominators for verification, reproducibility, and build simplicity, at > the expense of not being fully representative of the original build > capabilities and likely an irreversible step. > > I know there's been a lot of discussion here, and I probably missed it, > but early on we talked about things like Cython and cffi being part of > sdist generation (in setuptools at least). > > Are these things now expected to be deferred to the wheel building stage, > thus adding deps to the final build server, or are we just saying there is > less utility in supporting an explicit "artifacts directory" during sdist > generation? > Just that there is less utility in supporting a separate artifacts directory during sdist generation, correct. If the latter (and I think this is the case), that does seem reasonable, I > just wanted to confirm that such "compliations" are still expected, or at > least permissible, in downstream redistributables (sdists). > Personally if you want to do I want to help make it easier to do. -- > > C Anthony > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On Mon, Jul 10, 2017, at 04:13 PM, Nick Coghlan wrote: > My apologies folks, this is an entirely irrelevant tangent brought on > by my attempting to explain my own preference that source archives > (including sdists) actually *be* source archives, containing solely > the original software is *its preferred form for modification*. I also agree with this view - ideally, I think there should be no generated files in an sdist besides the metadata needed for tools consuming it. Many packages have included generated files to simplify building from an sdist, but I see this as a symptom of poor infrastructure which we're gradually fixing. In particular, before we had wheels, installing a package nearly always meant building it, so there was a strong pressure to do part of the build process before distributing the source. I don't think it's practical to forbid generating files to put in the sdist, but for pep517 I'd say it's appropriate to gently discourage it by not providing a build directory to that hook. I'm not going to argue hard for this if other people think the sdist hook needs that parameter, though. Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 10 July 2017 at 21:28, Thomas Kluyver wrote: > I don't think it's practical to forbid generating files to put in the > sdist, but for pep517 I'd say it's appropriate to gently discourage it > by not providing a build directory to that hook. I'm not going to argue > hard for this if other people think the sdist hook needs that parameter, > though. I also don't think it's a huge issue either way (and it's definitely a tangential issue as far as PEP 517 is concerned) but anecdotally I have encountered packages that have taken a substantial amount of time (30 minutes+) to run Cython on the sources[1]. Having pregenerated C files in the sdist made the difference between "usable" and "not worth bothering" in that case. Paul [1] It may be that was a Cython bug - but that's not actually relevant, as I didn't know enough to confirm it was a bug, nor did I know how to avoid triggering it. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A possible refactor/streamlining of PEP 517
On 11 July 2017 at 06:56, Paul Moore wrote: > On 10 July 2017 at 21:28, Thomas Kluyver wrote: >> I don't think it's practical to forbid generating files to put in the >> sdist, but for pep517 I'd say it's appropriate to gently discourage it >> by not providing a build directory to that hook. I'm not going to argue >> hard for this if other people think the sdist hook needs that parameter, >> though. > > I also don't think it's a huge issue either way (and it's definitely a > tangential issue as far as PEP 517 is concerned) but anecdotally I > have encountered packages that have taken a substantial amount of time > (30 minutes+) to run Cython on the sources[1]. Having pregenerated C > files in the sdist made the difference between "usable" and "not worth > bothering" in that case. Right, I realised through this discussion that my previous mental mapping of "sdist->SRPM, wheel->RPM" isn't actually *quite* right, since the objectives of the Python level packaging ecosystem and a redistributor-centric system like RPM aren't exactly the same. In particular, I now realise that if a redistributor wants to absolutely *ensure* that we have the source in the preferred form for modification, then we technically need to be starting with the VCS commit or release tarball rather than the sdist, and it's just an artifact of history that those two starting points have traditionally been roughly equivalent. This is due to the fact that it makes sense for publishers to optimise their sdists and wheels to provide the best possible experience for folks using the *Python* level tooling, whereby: - sdists are nominally architecture, platform and Python version independent source archives for common compile toolchains - wheels are pre-built archives for common Python versions, platforms and architectures So while I still suggest we omit "build_directory" from the build_sdist signature (at least for the initial iteration of the API design), I'm now only -0 on the idea, rather than my original -1. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
