[Distutils]Re: PEP 518 - pyproject.toml with no build-system.requires
On Sat, Jul 7, 2018 at 3:41 AM, Paul Moore wrote: > On 30 June 2018 at 06:33, Nick Coghlan wrote: >> On 28 June 2018 at 11:37, Nathaniel Smith wrote: >>> So my inclination is to plan on ending up with build-system.requires >>> defaulting to ["setuptools", "wheel"], and build-system.backend >>> defaulting to "setuptools". Hopefully we'll eventually get to a place >>> where ~no-one uses these defaults, but carrying around the code to >>> handle the defaults isn't really a burden. >> >> While I was going to post to say I liked this approach, after a bit of >> reflection, I realised I prefer Thomas Kluyver's suggestion: instead >> of taking "pyproject.toml" as indicating a build-isolation compatible >> sdist file, instead make "pyproject.toml with a build-system table" >> the marker for that case. > > As far as I can see, the only difference this makes is that it means > pip retains the legacy (non-isolated) behaviour in a few more places > (specifically places where it's quite likley the project hasn't > thought about build isolation). So it's basically a slightly more > forgiving version of Nathaniel's proposal. > > The part of Nathaniel's approach that I think would be most confusing > is a project that currently uses setup_requires which adds a > pyproject.toml for (say) towncrier. The build would become isolated, > but setup_requires (which is implemented by setuptools, not pip) would > ignore the isolated environment and install in the wrong place (maybe? > I honestly don't know). I'm quite happy to call this deprecated > behaviour and point out that the project should switch to explicitly > using PEP 518, but given that this whole discussion is because people > haven't done that, I suspect Nathaniel's proposal doesn't actually > solve the root issue here... Re: the interaction of build isolation and setup_requires: it looks like this is totally fine, actually. Based on some experiments + checking the docs, it appears that setup_requires has always done some magic where it doesn't actually try to install the requested packages into the current environment, but instead drops them inside the build directory and then uses some import system tricks to "overlay" them onto the current process's python path. So build isolation + setup_requires actually work very well together. I think in the long run, we want to enable build isolation everywhere. Packages that break when installed with build isolation are already broken when running 'pip install' in a fresh virtualenv. There probably are a few of these out there still that say things like "before installing this package, please install these other packages, as a separate call to pip", but it's been a long time now since I've seen one of those. And since they're already asking users to follow some finicky manual install procedure, requiring --no-build-isolation isn't a big deal. So, I don't care that much about what we use to trigger build isolation mode, because it's only a temporary thing anyway. The value of keying off something involving pyproject.toml is that it automatically gives us a kind of soft rollout: people adopting pyproject.toml are probably more willing to put up with issues with new packaging features, so we can hopefully shake out any problems before it becomes the standard. This suggests that our decision should be based on: if we want to be relatively more aggressive about rolling out build isolation, then we should key on the existence of pyproject.toml. If we want to be relatively more conservative, then we should key on the existence of build-system.requires. >> If you don't have a build-system table at all, then you'll continue to >> get the legacy sdist handling, allowing the addition of other tool >> config without impacting the way your sdist gets built. >> >> If you do add a build-system table, then you have to populates the >> "requires" field properly, even if you're using setuptools as your >> build backend. >> >> That way, the "build-system.backend defaults to setuptools" behaviour >> is only there to support pyproject.toml files that have already opted >> in to build isolation by writing: >> >> [build-system] >> requires = ["setuptools", "wheel"] >> > > That sounds fair to me. In terms of PEP wording: > > 1. build-system.requires becomes *optional* in pyproject.toml > 2. Tools should process projects without pyproject.toml in the same > way as they always have (backward compatibility). For pip, that means > no build isolation, and the old-style processing path. > 3. Tools should treat projects with pyproject.toml, but with *no* > build-system.requires key the same way as (2). > 4. Tools can assume that no legacy behaviour is needed for projects > that specify pyproject.toml and build-system.requires. > > Moving forward to PEP 517, we'd allow a default for > build-system.backend purely as a convenience because PEP 518 was > implemented before PEP 517 - but there's no intention or commitment to > retain *current*
[Distutils]Re: PEP 518 - pyproject.toml with no build-system.requires
On Sat, 7 Jul 2018, 16:12 Paul Moore, wrote: > On 30 June 2018 at 06:33, Nick Coghlan wrote: > > On 28 June 2018 at 11:37, Nathaniel Smith wrote: > >> So my inclination is to plan on ending up with build-system.requires > >> defaulting to ["setuptools", "wheel"], and build-system.backend > >> defaulting to "setuptools". Hopefully we'll eventually get to a place > >> where ~no-one uses these defaults, but carrying around the code to > >> handle the defaults isn't really a burden. > > > > While I was going to post to say I liked this approach, after a bit of > > reflection, I realised I prefer Thomas Kluyver's suggestion: instead > > of taking "pyproject.toml" as indicating a build-isolation compatible > > sdist file, instead make "pyproject.toml with a build-system table" > > the marker for that case. > > As far as I can see, the only difference this makes is that it means > pip retains the legacy (non-isolated) behaviour in a few more places > (specifically places where it's quite likley the project hasn't > thought about build isolation). So it's basically a slightly more > forgiving version of Nathaniel's proposal. > > The part of Nathaniel's approach that I think would be most confusing > is a project that currently uses setup_requires which adds a > pyproject.toml for (say) towncrier. The build would become isolated, > but setup_requires (which is implemented by setuptools, not pip) would > ignore the isolated environment and install in the wrong place (maybe? > I honestly don't know). I'm quite happy to call this deprecated > behaviour and point out that the project should switch to explicitly > using PEP 518, but given that this whole discussion is because people > haven't done that, I suspect Nathaniel's proposal doesn't actually > solve the root issue here... > > > If you don't have a build-system table at all, then you'll continue to > > get the legacy sdist handling, allowing the addition of other tool > > config without impacting the way your sdist gets built. > > > > If you do add a build-system table, then you have to populates the > > "requires" field properly, even if you're using setuptools as your > > build backend. > > > > That way, the "build-system.backend defaults to setuptools" behaviour > > is only there to support pyproject.toml files that have already opted > > in to build isolation by writing: > > > > [build-system] > > requires = ["setuptools", "wheel"] > > > > That sounds fair to me. In terms of PEP wording: > > 1. build-system.requires becomes *optional* in pyproject.toml > 2. Tools should process projects without pyproject.toml in the same > way as they always have (backward compatibility). For pip, that means > no build isolation, and the old-style processing path. > 3. Tools should treat projects with pyproject.toml, but with *no* > build-system.requires key the same way as (2). > 4. Tools can assume that no legacy behaviour is needed for projects > that specify pyproject.toml and build-system.requires. > > Moving forward to PEP 517, we'd allow a default for > build-system.backend purely as a convenience because PEP 518 was > implemented before PEP 517 - but there's no intention or commitment to > retain *current* PEP 518 code paths once PEP 517 is implemented (i.e, > nobody's suggesting that `build-system.backend missing` should *ever* > be different from `build-system.backend = "setuptools"`). > > Any objections? Specifically Brett made the point that this means that as a community we're OK with pyproject.toml being the standard > location for tool configuration, and not just for specifying build > tools. I guess I'm personally OK with this (although I do feel that > it's something we didn't fully talk through when writing the PEP, and > we're now getting pushed down this route by circumstance). It might > warrant a change to the PEP title, just to clarify the modified > intent. > > Nathaniel - are you happy with this variant rather than the one you > proposed? > > Is someone going to prepare PRs for the PEP and for pip to match this > behaviour? It sounds important enough that it should probably go into > pip 18.0 (although that's a discussion we can have on the pip tracker > - I'm not sure there's much we need to do, and there may be a PR > already). > I'll be happy to update pip for this after the PEP update. It's probably gonna be updating a conditional or so. :) > Paul > > PS For completeness, here's the range of options. Shout if I > misrepresented anything. > > pyproject.tomlbuild-system.requiresBehaviour > ---- > Does not existN/A Legacy > ExistsDoes not exist Current: Invalid file (error) >Thomas: Legacy >Nathaniel: Isolated > (setuptools, wheel) > ExistsExists Isolated (full PEP 518) > The pip 10 behavior is
[Distutils]Re: PEP 518 - pyproject.toml with no build-system.requires
On Sat, 7 Jul 2018 at 03:42 Paul Moore wrote: > On 30 June 2018 at 06:33, Nick Coghlan wrote: > > On 28 June 2018 at 11:37, Nathaniel Smith wrote: > >> So my inclination is to plan on ending up with build-system.requires > >> defaulting to ["setuptools", "wheel"], and build-system.backend > >> defaulting to "setuptools". Hopefully we'll eventually get to a place > >> where ~no-one uses these defaults, but carrying around the code to > >> handle the defaults isn't really a burden. > > > > While I was going to post to say I liked this approach, after a bit of > > reflection, I realised I prefer Thomas Kluyver's suggestion: instead > > of taking "pyproject.toml" as indicating a build-isolation compatible > > sdist file, instead make "pyproject.toml with a build-system table" > > the marker for that case. > > As far as I can see, the only difference this makes is that it means > pip retains the legacy (non-isolated) behaviour in a few more places > (specifically places where it's quite likley the project hasn't > thought about build isolation). So it's basically a slightly more > forgiving version of Nathaniel's proposal. > > The part of Nathaniel's approach that I think would be most confusing > is a project that currently uses setup_requires which adds a > pyproject.toml for (say) towncrier. The build would become isolated, > but setup_requires (which is implemented by setuptools, not pip) would > ignore the isolated environment and install in the wrong place (maybe? > I honestly don't know). I'm quite happy to call this deprecated > behaviour and point out that the project should switch to explicitly > using PEP 518, but given that this whole discussion is because people > haven't done that, I suspect Nathaniel's proposal doesn't actually > solve the root issue here... > > > If you don't have a build-system table at all, then you'll continue to > > get the legacy sdist handling, allowing the addition of other tool > > config without impacting the way your sdist gets built. > > > > If you do add a build-system table, then you have to populates the > > "requires" field properly, even if you're using setuptools as your > > build backend. > > > > That way, the "build-system.backend defaults to setuptools" behaviour > > is only there to support pyproject.toml files that have already opted > > in to build isolation by writing: > > > > [build-system] > > requires = ["setuptools", "wheel"] > > > > That sounds fair to me. In terms of PEP wording: > > 1. build-system.requires becomes *optional* in pyproject.toml > 2. Tools should process projects without pyproject.toml in the same > way as they always have (backward compatibility). For pip, that means > no build isolation, and the old-style processing path. > 3. Tools should treat projects with pyproject.toml, but with *no* > build-system.requires key the same way as (2). > 4. Tools can assume that no legacy behaviour is needed for projects > that specify pyproject.toml and build-system.requires. > > Moving forward to PEP 517, we'd allow a default for > build-system.backend purely as a convenience because PEP 518 was > implemented before PEP 517 - but there's no intention or commitment to > retain *current* PEP 518 code paths once PEP 517 is implemented (i.e, > nobody's suggesting that `build-system.backend missing` should *ever* > be different from `build-system.backend = "setuptools"`). > > Any objections? I do have a question about whether we specifically want the existence of build-system.requires or the existence of the entire build-system section to trigger the fallback behaviour? Is there a situation where any PEP-defined key in that section makes sense without 'requires'? IOW should having a build-system section imply the requirement of build-system.requires and leaving that key out is an error, while leaving the entire section out is now optional and is what triggers the fallback behaviour? I personally vote to say that if you define build-system you should go all-in and those it's an error to be missing 'requires' in that case. > Specifically Brett made the point that this means that > as a community we're OK with pyproject.toml being the standard > location for tool configuration, and not just for specifying build > tools. I guess I'm personally OK with this (although I do feel that > it's something we didn't fully talk through when writing the PEP, and > we're now getting pushed down this route by circumstance). It might > warrant a change to the PEP title, just to clarify the modified > intent. > Yep, the PEP would need a quick once-over to make sure the wording makes sense if this is the final decision, new packaging BDFL. ;) > > Nathaniel - are you happy with this variant rather than the one you > proposed? > > Is someone going to prepare PRs for the PEP and for pip to match this > behaviour? I can handle any PEP updates once we have settled and the exact change we want. > It sounds important enough that it should probably go
[Distutils]Re: Handing over default BDFL-Delegate responsibilities for packaging interoperability PEPs to Paul Moore
Thanks Nick and Paul!
Are there a few links which best describe the state of python packaging
interoperability PEPs?
https://www.pypa.io/en/latest/
https://www.pypa.io/en/latest/roadmap/
https://packaging.python.org/
https://github.com/pypa/interoperability-peps
https://github.com/pypa/interoperability-peps/issues
Are there any things you think should be prioritized going forward?
On Friday, July 6, 2018, Nathaniel Smith wrote:
> Nick, thanks so much for your service in an often thankless job. It is
> appreciated! And Paul, thanks for taking this on!
>
> On Fri, Jul 6, 2018, 19:08 Nick Coghlan wrote:
>
>> Hi folks,
>>
>> Since 2013, I've been the default BDFL-Delegate for packaging
>> interoperability PEPs. In that time, the Python packaging ecosystem
>> has moved forward in a lot of different areas, with pip being shipped
>> by default with CPython, the wheel binary packaging format reaching
>> ever-increasing heights of popularity, the cross-distro manylinux ABI
>> compatibility specification being developed, the new pyproject.toml
>> based sdist format being defined, the PSF's Packaging Working Group
>> being formed, the Python Packaging User Guide being developed, and
>> various aspects of the packaging metadata being enhanced to improve
>> the general user experience of the Python packaging ecosystem.
>>
>> The role of the BDFL-Delegate in that process is partly about making
>> arbitrary decisions when arbitrary decisions need to be made ("The
>> bikeshed shall be green!"), but also about helping to guide
>> discussions in productive directions, as well as determining when more
>> complex PEP level proposals have reached a sufficient level of
>> consensus that it makes sense to provisionally accept them and move on
>> to publishing reference implementations.
>>
>> While it's been a fascinating ~5 years, I've decided that it's time
>> for me to hand over those responsibilities to another PyPA
>> contributor. With Guido's approval, I've asked Paul Moore if he'd be
>> willing to take on the role, and Paul has graciously accepted the
>> additional responsibility.
>>
>> Paul's a long term pip contributor, and also a CPython core developer,
>> with a lot of practical experience in getting Python (and Python
>> packaging) to work well in Windows environments. He's also a familiar,
>> calm, and constructive presence in design discussions within
>> distutils-sig, pip and other PyPA projects, which is an important
>> characteristic when taking on BDFL-Delegate responsibilities.
>>
>> I'd like to personally thank Paul for being willing to take on this
>> task, and I look forward to many more productive design discussions!
>>
>> Cheers,
>> Nick.
>>
>> P.S. I'm not stepping down from Python packaging related activities
>> entirely, as I'll still be involved in Python Packaging User Guide and
>> pipenv maintenance, and will continue as a member of the PSF's
>> Packaging Working Group. However, the final sign-off for packaging
>> interoperability PEPs will now rest with Paul or someone else that
>> he appoints, rather than with me :)
>>
>> --
>> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
>> --
>> Distutils-SIG mailing list -- distutils-sig@python.org
>> To unsubscribe send an email to distutils-sig-le...@python.org
>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>> Message archived at https://mail.python.org/mm3/
>> archives/list/distutils-sig@python.org/message/
>> QT7SKORCF6OKWO3OVP5KO6XNGU2AR6TU/
>>
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/MSQ4XAFSZ4N5KZVM3LQJYS74U4RV7TMS/
[Distutils]Re: PEP 518 - pyproject.toml with no build-system.requires
On 30 June 2018 at 06:33, Nick Coghlan wrote: > On 28 June 2018 at 11:37, Nathaniel Smith wrote: >> So my inclination is to plan on ending up with build-system.requires >> defaulting to ["setuptools", "wheel"], and build-system.backend >> defaulting to "setuptools". Hopefully we'll eventually get to a place >> where ~no-one uses these defaults, but carrying around the code to >> handle the defaults isn't really a burden. > > While I was going to post to say I liked this approach, after a bit of > reflection, I realised I prefer Thomas Kluyver's suggestion: instead > of taking "pyproject.toml" as indicating a build-isolation compatible > sdist file, instead make "pyproject.toml with a build-system table" > the marker for that case. As far as I can see, the only difference this makes is that it means pip retains the legacy (non-isolated) behaviour in a few more places (specifically places where it's quite likley the project hasn't thought about build isolation). So it's basically a slightly more forgiving version of Nathaniel's proposal. The part of Nathaniel's approach that I think would be most confusing is a project that currently uses setup_requires which adds a pyproject.toml for (say) towncrier. The build would become isolated, but setup_requires (which is implemented by setuptools, not pip) would ignore the isolated environment and install in the wrong place (maybe? I honestly don't know). I'm quite happy to call this deprecated behaviour and point out that the project should switch to explicitly using PEP 518, but given that this whole discussion is because people haven't done that, I suspect Nathaniel's proposal doesn't actually solve the root issue here... > If you don't have a build-system table at all, then you'll continue to > get the legacy sdist handling, allowing the addition of other tool > config without impacting the way your sdist gets built. > > If you do add a build-system table, then you have to populates the > "requires" field properly, even if you're using setuptools as your > build backend. > > That way, the "build-system.backend defaults to setuptools" behaviour > is only there to support pyproject.toml files that have already opted > in to build isolation by writing: > > [build-system] > requires = ["setuptools", "wheel"] > That sounds fair to me. In terms of PEP wording: 1. build-system.requires becomes *optional* in pyproject.toml 2. Tools should process projects without pyproject.toml in the same way as they always have (backward compatibility). For pip, that means no build isolation, and the old-style processing path. 3. Tools should treat projects with pyproject.toml, but with *no* build-system.requires key the same way as (2). 4. Tools can assume that no legacy behaviour is needed for projects that specify pyproject.toml and build-system.requires. Moving forward to PEP 517, we'd allow a default for build-system.backend purely as a convenience because PEP 518 was implemented before PEP 517 - but there's no intention or commitment to retain *current* PEP 518 code paths once PEP 517 is implemented (i.e, nobody's suggesting that `build-system.backend missing` should *ever* be different from `build-system.backend = "setuptools"`). Any objections? Specifically Brett made the point that this means that as a community we're OK with pyproject.toml being the standard location for tool configuration, and not just for specifying build tools. I guess I'm personally OK with this (although I do feel that it's something we didn't fully talk through when writing the PEP, and we're now getting pushed down this route by circumstance). It might warrant a change to the PEP title, just to clarify the modified intent. Nathaniel - are you happy with this variant rather than the one you proposed? Is someone going to prepare PRs for the PEP and for pip to match this behaviour? It sounds important enough that it should probably go into pip 18.0 (although that's a discussion we can have on the pip tracker - I'm not sure there's much we need to do, and there may be a PR already). Paul PS For completeness, here's the range of options. Shout if I misrepresented anything. pyproject.tomlbuild-system.requiresBehaviour ---- Does not existN/A Legacy ExistsDoes not exist Current: Invalid file (error) Thomas: Legacy Nathaniel: Isolated (setuptools, wheel) ExistsExists Isolated (full PEP 518) -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/5DIB5C6KTBGRMLJ5H23UEJSZW5TVE2EA/
