Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

Sounds good, I'll bring it up there.

On Thu, Jun 29, 2017, 17:54 Bill Deegan  wrote:

> Daniel,
>
> Perhaps the better way to go about it than to call scons N times is to
> have N variant dirs? where there's a variant dir for each build type you
> want, and then youd call
>
> scons build_wheel
> which is aliased Alias('build_wheel','variant_dir for building wheel path')
>
> Maybe bring this up on scons users mailing list and we'll see if it can be
> resolved?
>
> On Wed, Jun 28, 2017 at 6:42 PM, Daniel Holth  wrote:
>
>> I was able to implement PEP 517 build_wheel and build_sdist for enscons
>> (on bitbucket), and a click cli calling any backend mentioned in
>> pyproject.toml. Pretty simple. Not sure what to do with the config
>> dictionary.
>>
>> SCons is not designed to be called twice in the same process with
>> different arguments. It would be easier for me to know that pip would only
>> invoke one PEP 517 function per subprocess, or to know all target
>> directories in advance. Otherwise the enscons.api subprocess has to invoke
>> SCons in another subprocess. SCons also builds all targets (wheel and sdist
>> in same invocation) by default.
>>
>> "Alakazam!"
>>
>> On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver 
>> wrote:
>>
>>> On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:
>>>
>>> Is there a prototype implementation of pep 517 yet?
>>>
>>>
>>> - Flit has a PR with a prototype backend implementation, though it's not
>>> up to date with all the changes the PEP has undergone. I'll update it when
>>> we've agreed on a spec - it's still a fast moving target right now.
>>> - I have a prototype module frontends could use to call hooks here:
>>> https://github.com/takluyver/pep517 . It's mostly up to date, except
>>> for the issue with using sdist as a fallback for copying files for a wheel.
>>>
>>> Re: magic strings - like Nathaniel said, I haven't noticed them as part
>>> of any proposal so far.
>>>
>>> Thomas
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Bill Deegan]

Daniel,

Perhaps the better way to go about it than to call scons N times is to have
N variant dirs? where there's a variant dir for each build type you want,
and then youd call

scons build_wheel
which is aliased Alias('build_wheel','variant_dir for building wheel path')

Maybe bring this up on scons users mailing list and we'll see if it can be
resolved?

On Wed, Jun 28, 2017 at 6:42 PM, Daniel Holth  wrote:

> I was able to implement PEP 517 build_wheel and build_sdist for enscons
> (on bitbucket), and a click cli calling any backend mentioned in
> pyproject.toml. Pretty simple. Not sure what to do with the config
> dictionary.
>
> SCons is not designed to be called twice in the same process with
> different arguments. It would be easier for me to know that pip would only
> invoke one PEP 517 function per subprocess, or to know all target
> directories in advance. Otherwise the enscons.api subprocess has to invoke
> SCons in another subprocess. SCons also builds all targets (wheel and sdist
> in same invocation) by default.
>
> "Alakazam!"
>
> On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver 
> wrote:
>
>> On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:
>>
>> Is there a prototype implementation of pep 517 yet?
>>
>>
>> - Flit has a PR with a prototype backend implementation, though it's not
>> up to date with all the changes the PEP has undergone. I'll update it when
>> we've agreed on a spec - it's still a fast moving target right now.
>> - I have a prototype module frontends could use to call hooks here:
>> https://github.com/takluyver/pep517 . It's mostly up to date, except for
>> the issue with using sdist as a fallback for copying files for a wheel.
>>
>> Re: magic strings - like Nathaniel said, I haven't noticed them as part
>> of any proposal so far.
>>
>> Thomas
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

The other thing I noticed is that build-system requires has no hyphen but
build-backend does.

On Wed, Jun 28, 2017, 22:28 Daniel Holth  wrote:

> That would fit with setup.py, which also likes to sys.exit() after one
> command.
>
> On Wed, Jun 28, 2017, 22:25 Nick Coghlan  wrote:
>
>> On 29 June 2017 at 11:42, Daniel Holth  wrote:
>> > I was able to implement PEP 517 build_wheel and build_sdist for enscons
>> (on
>> > bitbucket), and a click cli calling any backend mentioned in
>> pyproject.toml.
>> > Pretty simple. Not sure what to do with the config dictionary.
>>
>> That's designed to allow frontends to tunnel custom settings from the
>> user through to the backend, so if you don't support any config
>> settings yet, it probably makes sense to just error out with "Unknown
>> config setting" if the dictionary is non-empty. Alternatively, you
>> could silently ignore it.
>>
>> > SCons is not designed to be called twice in the same process with
>> different
>> > arguments. It would be easier for me to know that pip would only invoke
>> one
>> > PEP 517 function per subprocess, or to know all target directories in
>> > advance. Otherwise the enscons.api subprocess has to invoke SCons in
>> another
>> > subprocess. SCons also builds all targets (wheel and sdist in same
>> > invocation) by default.
>>
>> That's an interesting point of tension between supporting imperative
>> frontends like pip (which have a strong opinion about the order in
>> which steps should be executed) and declarative build systems like
>> Scons (which are more "produce the defined artifact set").
>>
>> However, I think the way to go for now would be to say:
>>
>> - each hook call should be made in a fresh subprocess (so backends
>> don't need to use a second subprocess "just to be on the safe side")
>> - in a *future* API extension, we may add an optional "build_all" hook
>> (whereby the backend produced both an sdist and all wheels it knew how
>> to create for the current platform), but anything like that will be
>> out of scope for the initial PEP 517 API
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

That would fit with setup.py, which also likes to sys.exit() after one
command.

On Wed, Jun 28, 2017, 22:25 Nick Coghlan  wrote:

> On 29 June 2017 at 11:42, Daniel Holth  wrote:
> > I was able to implement PEP 517 build_wheel and build_sdist for enscons
> (on
> > bitbucket), and a click cli calling any backend mentioned in
> pyproject.toml.
> > Pretty simple. Not sure what to do with the config dictionary.
>
> That's designed to allow frontends to tunnel custom settings from the
> user through to the backend, so if you don't support any config
> settings yet, it probably makes sense to just error out with "Unknown
> config setting" if the dictionary is non-empty. Alternatively, you
> could silently ignore it.
>
> > SCons is not designed to be called twice in the same process with
> different
> > arguments. It would be easier for me to know that pip would only invoke
> one
> > PEP 517 function per subprocess, or to know all target directories in
> > advance. Otherwise the enscons.api subprocess has to invoke SCons in
> another
> > subprocess. SCons also builds all targets (wheel and sdist in same
> > invocation) by default.
>
> That's an interesting point of tension between supporting imperative
> frontends like pip (which have a strong opinion about the order in
> which steps should be executed) and declarative build systems like
> Scons (which are more "produce the defined artifact set").
>
> However, I think the way to go for now would be to say:
>
> - each hook call should be made in a fresh subprocess (so backends
> don't need to use a second subprocess "just to be on the safe side")
> - in a *future* API extension, we may add an optional "build_all" hook
> (whereby the backend produced both an sdist and all wheels it knew how
> to create for the current platform), but anything like that will be
> out of scope for the initial PEP 517 API
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 29 June 2017 at 11:42, Daniel Holth  wrote:
> I was able to implement PEP 517 build_wheel and build_sdist for enscons (on
> bitbucket), and a click cli calling any backend mentioned in pyproject.toml.
> Pretty simple. Not sure what to do with the config dictionary.

That's designed to allow frontends to tunnel custom settings from the
user through to the backend, so if you don't support any config
settings yet, it probably makes sense to just error out with "Unknown
config setting" if the dictionary is non-empty. Alternatively, you
could silently ignore it.

> SCons is not designed to be called twice in the same process with different
> arguments. It would be easier for me to know that pip would only invoke one
> PEP 517 function per subprocess, or to know all target directories in
> advance. Otherwise the enscons.api subprocess has to invoke SCons in another
> subprocess. SCons also builds all targets (wheel and sdist in same
> invocation) by default.

That's an interesting point of tension between supporting imperative
frontends like pip (which have a strong opinion about the order in
which steps should be executed) and declarative build systems like
Scons (which are more "produce the defined artifact set").

However, I think the way to go for now would be to say:

- each hook call should be made in a fresh subprocess (so backends
don't need to use a second subprocess "just to be on the safe side")
- in a *future* API extension, we may add an optional "build_all" hook
(whereby the backend produced both an sdist and all wheels it knew how
to create for the current platform), but anything like that will be
out of scope for the initial PEP 517 API

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

I was able to implement PEP 517 build_wheel and build_sdist for enscons (on
bitbucket), and a click cli calling any backend mentioned in
pyproject.toml. Pretty simple. Not sure what to do with the config
dictionary.

SCons is not designed to be called twice in the same process with different
arguments. It would be easier for me to know that pip would only invoke one
PEP 517 function per subprocess, or to know all target directories in
advance. Otherwise the enscons.api subprocess has to invoke SCons in
another subprocess. SCons also builds all targets (wheel and sdist in same
invocation) by default.

"Alakazam!"

On Wed, Jun 28, 2017 at 3:52 PM Thomas Kluyver  wrote:

> On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:
>
> Is there a prototype implementation of pep 517 yet?
>
>
> - Flit has a PR with a prototype backend implementation, though it's not
> up to date with all the changes the PEP has undergone. I'll update it when
> we've agreed on a spec - it's still a fast moving target right now.
> - I have a prototype module frontends could use to call hooks here:
> https://github.com/takluyver/pep517 . It's mostly up to date, except for
> the issue with using sdist as a fallback for copying files for a wheel.
>
> Re: magic strings - like Nathaniel said, I haven't noticed them as part of
> any proposal so far.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

On Wed, Jun 28, 2017, at 06:07 PM, Daniel Holth wrote:
> Is there a prototype implementation of pep 517 yet?



- Flit has a PR with a prototype backend implementation, though it's
  not up to date with all the changes the PEP has undergone. I'll
  update it when we've agreed on a spec - it's still a fast moving
  target right now.- I have a prototype module frontends could use to call 
hooks here:
  https://github.com/takluyver/pep517 . It's mostly up to date,
  except for the issue with using sdist as a fallback for copying
  files for a wheel.
Re: magic strings - like Nathaniel said, I haven't noticed them as part of any 
proposal so far.
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

I don't think anyone has proposed anything involving magic strings?

On Jun 28, 2017 3:58 AM, "Robert Collins"  wrote:

>
>
> Re: returning magic strings to indicate exceptions. Please no. Just no.
> Have any pep build host add a small library to Python path with any symbols
> we want to define. Hardly an implementation hurdle.
>
> Rob
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

Is there a prototype implementation of pep 517 yet?

On Wed, Jun 28, 2017, 06:58 Robert Collins 
wrote:

>
>
> Re: returning magic strings to indicate exceptions. Please no. Just no.
> Have any pep build host add a small library to Python path with any symbols
> we want to define. Hardly an implementation hurdle.
>
> Rob
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Robert Collins]

Re: returning magic strings to indicate exceptions. Please no. Just no.
Have any pep build host add a small library to Python path with any symbols
we want to define. Hardly an implementation hurdle.

Rob
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Paul Moore]

On 28 June 2017 at 05:29, Nick Coghlan  wrote:
> On 28 June 2017 at 13:43, Nathaniel Smith  wrote:
>> I would say that there's only one required path, which is in-place.
>> (But the tree that they do the in-place build in might have a more or
>> less complicated history). It worries me that you seem to think that
>> build_sdist and build_wheel should be coupled like this... this
>> proliferation of cases and the reification of "out of tree builds" as
>> a special thing that's different from an in-tree build in a temporary
>> tree is already what worried me about prepare_wheel_input_files, and
>> now it seems to be spreading :-).
>
> I don't know what additional coupling you're seeing: the only coupling
> is that building a wheel directly from a VCS source tree and first
> exporting an sdist and then building a wheel from that *must give the
> same result* (modulo any problems related to reproducible builds, or
> the lack thereof). If a project can't meet that expectation, then
> their sdist generation is broken, since it clearly isn't exporting the
> necessary content to actually build the project properly.
>
> If you weren't aware of that inherent coupling, and are surprised by
> the fact that it's a constraint on publishing useful sdists, then I'd
> consider it a good thing that PEP 517 is now making it more explicit.

Note that the whole concept of "out of tree builds" is not something
theoretical - it comes directly from pip's requirement to be able to
do clean builds. So saying that tree->sdist->unpacked sdist is "just
another in-place build of a different tree" is missing the point,
which is that we need a mechanism to produce an effectively identical,
in the sense that it must result in the same wheel, copy of a tree -
and because we chose the sdist as the route for that, there are
constraints on the contents of a sdist.

I view those constraints as pretty much self evident, as I would be
extremely surprised if I had a source tree of *any* form for a
project, and I build a sdist from it and gave it to a friend, and when
we both built wheels and installed them, we had different installed
copies of the project.

The alternative would be to have a separate "produce a copy of this
tree that builds the same wheel" hook. And I thought we already went
round that loop, which is how we ended up deciding that the sdist was
our approach...

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 28 June 2017 at 14:36, Nathaniel Smith  wrote:
> I think that in these two cases, all the hooks used in the
> wheel-building phase should be run in exactly the same way, i.e.,
> inside the unpacked sdist, with no reference to the original working
> tree that that sdist was generated from. Since the PyPI case can't
> access the original working tree, the local out-of-place-build case
> generate the sdist and then after that pretend it can't access the
> original working tree either.

That's up to the frontend - they're certainly free to do things that
way, for the reasons you give. However, backends aren't allowed to
assume that *all* frontends will work that way - they need to ensure
that the "query-VCS-tree, build-from-sdist" combination works sensibly
(and most of them will, through the simple expedient of not reporting
any dynamic build dependencies in the first place).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 28 June 2017 at 14:25, Nathaniel Smith  wrote:
> pyproject.toml is just an example anyway; get_build_wheel_requires
> runs arbitrary code, and that code could potentially return different
> results in different trees.

No, I don't want to give backends permission to work that way. If a
frontend runs get_build_wheel_requires() against the VCS tree, then
that should also be sufficient to set up an environment to build an
unpacked sdist - the backend isn't permitted to require that both
hooks be run against the exact same tree. (It's fine if it returns
some extra dependencies that may not technically be needed for a
from-sdist build, though)

To put it another way, if the "VCS-tree-for-build-requirements,
unpacked-sdist-tree for build" combination doesn't work, that's a bug
in the backend, not a bug in the frontend.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Tue, Jun 27, 2017 at 9:29 PM, Nick Coghlan  wrote:
> Sorry, I also meant to reply to this part before hitting send.
>
> On 28 June 2017 at 13:43, Nathaniel Smith  wrote:
>> I would say that there's only one required path, which is in-place.
>> (But the tree that they do the in-place build in might have a more or
>> less complicated history). It worries me that you seem to think that
>> build_sdist and build_wheel should be coupled like this... this
>> proliferation of cases and the reification of "out of tree builds" as
>> a special thing that's different from an in-tree build in a temporary
>> tree is already what worried me about prepare_wheel_input_files, and
>> now it seems to be spreading :-).
>
> I don't know what additional coupling you're seeing: the only coupling
> is that building a wheel directly from a VCS source tree and first
> exporting an sdist and then building a wheel from that *must give the
> same result* (modulo any problems related to reproducible builds, or
> the lack thereof). If a project can't meet that expectation, then
> their sdist generation is broken, since it clearly isn't exporting the
> necessary content to actually build the project properly.
>
> If you weren't aware of that inherent coupling, and are surprised by
> the fact that it's a constraint on publishing useful sdists, then I'd
> consider it a good thing that PEP 517 is now making it more explicit.

What I'm concerned about is that in your formulation, you're running
different code to build a wheel from an sdist that we just generated,
versus an sdist that we downloaded from PyPI. By making it possible to
distinguish these cases, you're facilitating projects that break that
inherent coupling (either accidentally or on purpose).

I think that in these two cases, all the hooks used in the
wheel-building phase should be run in exactly the same way, i.e.,
inside the unpacked sdist, with no reference to the original working
tree that that sdist was generated from. Since the PyPI case can't
access the original working tree, the local out-of-place-build case
generate the sdist and then after that pretend it can't access the
original working tree either.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

Sorry, I also meant to reply to this part before hitting send.

On 28 June 2017 at 13:43, Nathaniel Smith  wrote:
> I would say that there's only one required path, which is in-place.
> (But the tree that they do the in-place build in might have a more or
> less complicated history). It worries me that you seem to think that
> build_sdist and build_wheel should be coupled like this... this
> proliferation of cases and the reification of "out of tree builds" as
> a special thing that's different from an in-tree build in a temporary
> tree is already what worried me about prepare_wheel_input_files, and
> now it seems to be spreading :-).

I don't know what additional coupling you're seeing: the only coupling
is that building a wheel directly from a VCS source tree and first
exporting an sdist and then building a wheel from that *must give the
same result* (modulo any problems related to reproducible builds, or
the lack thereof). If a project can't meet that expectation, then
their sdist generation is broken, since it clearly isn't exporting the
necessary content to actually build the project properly.

If you weren't aware of that inherent coupling, and are surprised by
the fact that it's a constraint on publishing useful sdists, then I'd
consider it a good thing that PEP 517 is now making it more explicit.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Tue, Jun 27, 2017 at 9:20 PM, Nick Coghlan  wrote:
> On 28 June 2017 at 13:43, Nathaniel Smith  wrote:
>> On Tue, Jun 27, 2017 at 4:20 AM, Nick Coghlan  wrote:
>>> On 27 June 2017 at 18:54, Nathaniel Smith  wrote:
 On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan  wrote:
> 2. If you don't find the input preparation hook:
>
> - call `get_build_sdist_requires()`
> - call `get_build_wheel_requires()`
> - install both sets of additional dependencies
> - call `build_sdist()`
> - unpack the sdist into the build directory
> - call `build_wheel()`

 Side note: I think this is wrong -- I think you can't call
 get_build_wheel_requires until after you've unpacked the sdist,
 because that's a whole new tree that might be arbitrarily different.
 (In practice I guess this should never happen, but we should be clear
 conceptually I think.)
>>>
>>> No, both `get_build_wheel_requires()` and `build_wheel()` also have to
>>> work in-place - backends aren't allowed to assume that *all* builds
>>> will be out-of-tree, even though pip specifically is expected to work
>>> that way. Neither of these hooks is allowed to give different results
>>> depending on whether you're doing an in-place or out-of-tree build.
>>
>> Right. But if you're building and sdist and unpacking it and then
>> calling build_wheel there, then that's not an out-of-tree build, it's
>> an in-tree build in a new tree. The wheel hooks are certainly allowed
>> to give different results in a VCS-tree and an unpacked-sdist-tree,
>> they have different pyproject.toml files!
>
> Umm, no. The pyproject.toml file MUST NOT change when building the
> sdist. It's a human-managed input file - if any step of the build
> process mutates it, that step is *broken*.

I agree that changing the pyproject.toml file is a terrible idea in most cases

I still think we should do the "right thing" when it happens, where
"right" means "doesn't involve complex underspecified coupling between
conceptually distinct phases", because this thing needs to fit in
people's brains. It's not so much that pyproject.toml changing is an
important case, as that if we can even *tell* whether it changed then
there's something wrong with our architecture.

pyproject.toml is just an example anyway; get_build_wheel_requires
runs arbitrary code, and that code could potentially return different
results in different trees.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 28 June 2017 at 13:43, Nathaniel Smith  wrote:
> On Tue, Jun 27, 2017 at 4:20 AM, Nick Coghlan  wrote:
>> On 27 June 2017 at 18:54, Nathaniel Smith  wrote:
>>> On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan  wrote:
 2. If you don't find the input preparation hook:

 - call `get_build_sdist_requires()`
 - call `get_build_wheel_requires()`
 - install both sets of additional dependencies
 - call `build_sdist()`
 - unpack the sdist into the build directory
 - call `build_wheel()`
>>>
>>> Side note: I think this is wrong -- I think you can't call
>>> get_build_wheel_requires until after you've unpacked the sdist,
>>> because that's a whole new tree that might be arbitrarily different.
>>> (In practice I guess this should never happen, but we should be clear
>>> conceptually I think.)
>>
>> No, both `get_build_wheel_requires()` and `build_wheel()` also have to
>> work in-place - backends aren't allowed to assume that *all* builds
>> will be out-of-tree, even though pip specifically is expected to work
>> that way. Neither of these hooks is allowed to give different results
>> depending on whether you're doing an in-place or out-of-tree build.
>
> Right. But if you're building and sdist and unpacking it and then
> calling build_wheel there, then that's not an out-of-tree build, it's
> an in-tree build in a new tree. The wheel hooks are certainly allowed
> to give different results in a VCS-tree and an unpacked-sdist-tree,
> they have different pyproject.toml files!

Umm, no. The pyproject.toml file MUST NOT change when building the
sdist. It's a human-managed input file - if any step of the build
process mutates it, that step is *broken*.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Tue, Jun 27, 2017 at 4:20 AM, Nick Coghlan  wrote:
> On 27 June 2017 at 18:54, Nathaniel Smith  wrote:
>> On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan  wrote:
>>> 2. If you don't find the input preparation hook:
>>>
>>> - call `get_build_sdist_requires()`
>>> - call `get_build_wheel_requires()`
>>> - install both sets of additional dependencies
>>> - call `build_sdist()`
>>> - unpack the sdist into the build directory
>>> - call `build_wheel()`
>>
>> Side note: I think this is wrong -- I think you can't call
>> get_build_wheel_requires until after you've unpacked the sdist,
>> because that's a whole new tree that might be arbitrarily different.
>> (In practice I guess this should never happen, but we should be clear
>> conceptually I think.)
>
> No, both `get_build_wheel_requires()` and `build_wheel()` also have to
> work in-place - backends aren't allowed to assume that *all* builds
> will be out-of-tree, even though pip specifically is expected to work
> that way. Neither of these hooks is allowed to give different results
> depending on whether you're doing an in-place or out-of-tree build.

Right. But if you're building and sdist and unpacking it and then
calling build_wheel there, then that's not an out-of-tree build, it's
an in-tree build in a new tree. The wheel hooks are certainly allowed
to give different results in a VCS-tree and an unpacked-sdist-tree,
they have different pyproject.toml files!

> To put it another way, there are two build paths that backends are
> required to support:
>
> - in-place, direct from the original source tree
> - out-of-tree, from an unpacked sdist
>
> There's also a third, optional, build path for backends that define
> `prepare_wheel_input_files`:
>
> - out-of-tree, from a prepared wheel input directory

I would say that there's only one required path, which is in-place.
(But the tree that they do the in-place build in might have a more or
less complicated history). It worries me that you seem to think that
build_sdist and build_wheel should be coupled like this... this
proliferation of cases and the reification of "out of tree builds" as
a special thing that's different from an in-tree build in a temporary
tree is already what worried me about prepare_wheel_input_files, and
now it seems to be spreading :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 27 June 2017 at 18:54, Nathaniel Smith  wrote:
> On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan  wrote:
>> 2. If you don't find the input preparation hook:
>>
>> - call `get_build_sdist_requires()`
>> - call `get_build_wheel_requires()`
>> - install both sets of additional dependencies
>> - call `build_sdist()`
>> - unpack the sdist into the build directory
>> - call `build_wheel()`
>
> Side note: I think this is wrong -- I think you can't call
> get_build_wheel_requires until after you've unpacked the sdist,
> because that's a whole new tree that might be arbitrarily different.
> (In practice I guess this should never happen, but we should be clear
> conceptually I think.)

No, both `get_build_wheel_requires()` and `build_wheel()` also have to
work in-place - backends aren't allowed to assume that *all* builds
will be out-of-tree, even though pip specifically is expected to work
that way. Neither of these hooks is allowed to give different results
depending on whether you're doing an in-place or out-of-tree build.

To put it another way, there are two build paths that backends are
required to support:

- in-place, direct from the original source tree
- out-of-tree, from an unpacked sdist

There's also a third, optional, build path for backends that define
`prepare_wheel_input_files`:

- out-of-tree, from a prepared wheel input directory

The result wheel should be "the same" regardless of how the build is
executed. In practice, we expect there will be some variation, just
because reproducible builds are hard. However, the output of the
different paths should be at least as similar as any two builds of the
same source tree run at different times with the same system
configuration.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Tue, Jun 27, 2017 at 12:27 AM, Nick Coghlan  wrote:
> On 26 June 2017 at 22:36, Nathaniel Smith  wrote:
>> On Jun 25, 2017 12:46 AM, "Nick Coghlan"  wrote:
>>
>> On 25 June 2017 at 17:41, Nathaniel Smith  wrote:
>>> Maybe you're right and there are exactly 2 front-end use cases and it
>>> will turn out that the current PEP addresses them perfectly. I don't
>>> have a crystal ball; I'm making an argument from ignorance.
>>
>> I'm not - we have two concrete potential consumers of the interface
>> (pip and tox, aka "build to use" and "build to test"), and I'm
>> designing the interface to cover their needs (i.e. out-of-tree wheel
>> builds and actual sdists).
>>
>> If we discover other use cases later, we'll worry about them then (and
>> the easy of doing so is the nicest benefit of defining this as a
>> Python API), but the temptation to design in hyper-flexibility now
>> falls under YAGNI (You Ain' Gonna Need It).
>>
>>
>> My proposal also covers their needs AFAICT?
>
> No, as you don't know in your proposal whether or not build_sdist can
> fail until after you've already called it, and you always need to call
> both `get_build_sdist_requires()` and `build_sdist()`, even if you
> only care about doing an out-of-tree wheel build.

Ah, this is an interesting point, thank you! I hadn't thought of that.

But... it doesn't seem like a big deal in practice? If a build backend
knows that it can't build an sdist from a given tree, then its
get_build_sdist_requires hook (which is arbitrary code after all) can
surely figure this out and return []. Or we could allow
get_build_sdist_requires to also return NotImplemented (defining that
to mean "building sdists is not implemented, don't even try calling
build_sdist"), though maybe that's unnecessary fiddliness.

> 2. If you don't find the input preparation hook:
>
> - call `get_build_sdist_requires()`
> - call `get_build_wheel_requires()`
> - install both sets of additional dependencies
> - call `build_sdist()`
> - unpack the sdist into the build directory
> - call `build_wheel()`

Side note: I think this is wrong -- I think you can't call
get_build_wheel_requires until after you've unpacked the sdist,
because that's a whole new tree that might be arbitrarily different.
(In practice I guess this should never happen, but we should be clear
conceptually I think.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 27 June 2017 at 01:33, Daniel Holth  wrote:
> My impressions on reading what is hopefully the current version of the PEP
>
> Are there different paths to get to sdist/wheel for example tree -> prepare
> sdist/wheel files -> build sdist and tree -> build sdist, tree -> build
> wheel depending on what [pip] decides to do? Should the frontend do both and
> compare the result to make sure it is the same?
>
> I'm not entirely clear on what the prepare hooks should do.
>
> The rationalizations interleaved between the descriptions of each hook
> distract from what the hooks should actually do. Suggested replacement
> rationalization: "Some people like to build [format]. This hook builds
> [format]".

I think it would be worth restructuring that part of the PEP so the
required hooks (build_sdist, build_wheel) are listed in their own
section, followed by a separate section for the optional ones
(get_build_sdist_requires, get_build_wheel_requires,
prepare_wheel_metadata, prepare_wheel_input_files)

The latter four all already define what frontends are expected to do
if they're missing:

get_build_sdist_requires: assume no extra dependencies for build_sdist
get_build_wheel_requires: assume no extra dependencies for build_wheel
prepare_wheel_metadata: call build_wheel and extract the dist-info directory
prepare_wheel_input_files: call build_sdist and unpack the resulting archive

Backends will only need to implement those if the default behaviours
are either wrong (e.g. the backend may need extra platform dependent
dependencies), or grossly inefficient.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 26 June 2017 at 22:36, Nathaniel Smith  wrote:
> On Jun 25, 2017 12:46 AM, "Nick Coghlan"  wrote:
>
> On 25 June 2017 at 17:41, Nathaniel Smith  wrote:
>> Maybe you're right and there are exactly 2 front-end use cases and it
>> will turn out that the current PEP addresses them perfectly. I don't
>> have a crystal ball; I'm making an argument from ignorance.
>
> I'm not - we have two concrete potential consumers of the interface
> (pip and tox, aka "build to use" and "build to test"), and I'm
> designing the interface to cover their needs (i.e. out-of-tree wheel
> builds and actual sdists).
>
> If we discover other use cases later, we'll worry about them then (and
> the easy of doing so is the nicest benefit of defining this as a
> Python API), but the temptation to design in hyper-flexibility now
> falls under YAGNI (You Ain' Gonna Need It).
>
>
> My proposal also covers their needs AFAICT?

No, as you don't know in your proposal whether or not build_sdist can
fail until after you've already called it, and you always need to call
both `get_build_sdist_requires()` and `build_sdist()`, even if you
only care about doing an out-of-tree wheel build.

That categorically rules out two-pass introspection based
implementations that first interrogate the backend to find out which
hooks it supports, and then use that to decide their execution
strategy.

Tangent: it turns out the word order difference between
`get_build_wheel_requires` and `prepare_wheel_build_files` hurts my
brain, so I've changed the latter to `prepare_wheel_input_files`
below. That keeps the common prefix with `prepare_wheel_metadata`,
while avoiding the confusing word order reversal relative to
`build_wheel`.

That is, the current PEP allows a frontend to do the following when
given an arbitrary source tree rather than an sdist archive:

1. Look for `prepare_wheel_input_files()` on the backend. If you find it:

- call `get_build_wheel_requires()`
- install the additional dependencies
- call `prepare_wheel_input_files()`
- call `build_wheel()`

2. If you don't find the input preparation hook:

- call `get_build_sdist_requires()`
- call `get_build_wheel_requires()`
- install both sets of additional dependencies
- call `build_sdist()`
- unpack the sdist into the build directory
- call `build_wheel()`

Failure of any hook in any way (whether that's raising an exception or
returning something that isn't in line with the requirements of PEP
517) indicates a failed build.

To put it another way, the key reason we can be confident that PEP 517
is comprehensive as currently drafted is that we only have two target
artifact types (sdist & wheel), one source archiving strategy
(in-tree), and two binary build strategies (in-tree or out-of-tree).

That means source archiving is straightforward:

- call `get_build_sdist_requires()`
- install the additional dependencies
- call `build_sdist()`

In-tree binary builds are also straightforward:

- call `get_build_wheel_requires()`
- install the additional dependencies
- call `build_wheel()`

Out-of-tree binary builds via sdist combine the two:

- call `get_build_sdist_requires()`
- call `get_build_wheel_requires()`
- install both sets of additional dependencies
- call `build_sdist()`
- unpack the sdist into the build directory
- call `build_wheel()`

(Frontends would also be free to use two distinct environments, one to
create the sdist, and then a separate one to build the wheel)

However, that last strategy has an undesirable property: it means that
installing the sdist archiving dependencies becomes a constraint on
out-of-tree builds, which is annoying for both frontends and backends.
Adding direct support for out-of-tree builds resolves that problem.

Out-of-tree binary builds via a dedicated hook:

- call `get_build_wheel_requires()`
- install the additional dependencies
- call `prepare_wheel_input_files()`
- call `build_wheel()`

By contrast, if we only define a "try it and see if it works" approach
for build_sdist, and provide no other way to request an out-of-tree
build, then frontends *have* to do the following:

- call `get_build_sdist_requires()`
- call `get_build_wheel_requires()`
- install both sets of additional dependencies
- call `build_sdist()`
- if it returns `NotImplemented`... umm, give up, same as if it
failed? Fall back to the bad "copy everything" default that pip is
trying to get away from?
- unpack the sdist into the build directory
- call `build_wheel()`

Without a dedicated out-of-tree build preparation hook, there's no way
for frontends to know in advance that build_sdist might fail, there's
no way for them to avoid installing the sdist archiving dependencies
when doing an out-of-tree build, and there's no way for backends to
provide a fallback build directory preparation strategy that relies
solely on the wheel building dependencies.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist

Re: [Distutils] Finishing up PEP 517

[Daniel Holth]

My impressions on reading what is hopefully the current version of the PEP

Are there different paths to get to sdist/wheel for example tree -> prepare
sdist/wheel files -> build sdist and tree -> build sdist, tree -> build
wheel depending on what [pip] decides to do? Should the frontend do both
and compare the result to make sure it is the same?

I'm not entirely clear on what the prepare hooks should do.

The rationalizations interleaved between the descriptions of each hook
distract from what the hooks should actually do. Suggested replacement
rationalization: "Some people like to build [format]. This hook builds
[format]".

I will make an effort to implement this for enscons, probably with the most
direct tree -> wheel, tree -> sdist hooks and without any optional hooks.

Thanks,

Daniel

On Mon, Jun 26, 2017 at 8:36 AM Nathaniel Smith  wrote:

> On Jun 25, 2017 12:46 AM, "Nick Coghlan"  wrote:
>
> On 25 June 2017 at 17:41, Nathaniel Smith  wrote:
> > Maybe you're right and there are exactly 2 front-end use cases and it
> > will turn out that the current PEP addresses them perfectly. I don't
> > have a crystal ball; I'm making an argument from ignorance.
>
> I'm not - we have two concrete potential consumers of the interface
> (pip and tox, aka "build to use" and "build to test"), and I'm
> designing the interface to cover their needs (i.e. out-of-tree wheel
> builds and actual sdists).
>
> If we discover other use cases later, we'll worry about them then (and
> the easy of doing so is the nicest benefit of defining this as a
> Python API), but the temptation to design in hyper-flexibility now
> falls under YAGNI (You Ain' Gonna Need It).
>
>
> My proposal also covers their needs AFAICT? At least I thought Donald said
> he thought the would work for pip. And you can't use YAGNI to argue for a
> more complicated proposal, that's cheating :-).
>
> -n
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Jun 25, 2017 12:46 AM, "Nick Coghlan"  wrote:

On 25 June 2017 at 17:41, Nathaniel Smith  wrote:
> Maybe you're right and there are exactly 2 front-end use cases and it
> will turn out that the current PEP addresses them perfectly. I don't
> have a crystal ball; I'm making an argument from ignorance.

I'm not - we have two concrete potential consumers of the interface
(pip and tox, aka "build to use" and "build to test"), and I'm
designing the interface to cover their needs (i.e. out-of-tree wheel
builds and actual sdists).

If we discover other use cases later, we'll worry about them then (and
the easy of doing so is the nicest benefit of defining this as a
Python API), but the temptation to design in hyper-flexibility now
falls under YAGNI (You Ain' Gonna Need It).


My proposal also covers their needs AFAICT? At least I thought Donald said
he thought the would work for pip. And you can't use YAGNI to argue for a
more complicated proposal, that's cheating :-).

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 25 June 2017 at 17:41, Nathaniel Smith  wrote:
> Maybe you're right and there are exactly 2 front-end use cases and it
> will turn out that the current PEP addresses them perfectly. I don't
> have a crystal ball; I'm making an argument from ignorance.

I'm not - we have two concrete potential consumers of the interface
(pip and tox, aka "build to use" and "build to test"), and I'm
designing the interface to cover their needs (i.e. out-of-tree wheel
builds and actual sdists).

If we discover other use cases later, we'll worry about them then (and
the easy of doing so is the nicest benefit of defining this as a
Python API), but the temptation to design in hyper-flexibility now
falls under YAGNI (You Ain' Gonna Need It).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

On Sun, Jun 25, 2017, at 08:41 AM, Nick Coghlan wrote:
> Aside from not producing an archive, the bit that makes
> `prepare_wheel_build_files` notably different from `build_sdist` is
> that it relies on `get_build_wheel_requires` *not* on
> `get_build_sdist_requires`:
> https://www.python.org/dev/peps/pep-0517/#build-environment

This reminds me: while implementing a wrapper to call PEP 517 hooks, a
problem occurred to me. I have implemented prepare_wheel_build_files
with a fallback to making an sdist if the hook is not defined:

https://github.com/takluyver/pep517/blob/ee43a9334c377d7c37badcc8527cb7a8500180f7/pep517/_in_process.py#L136

But in this case, the build_sdist hook will be invoked in an environment
with the build-wheel-deps installed, not the build-sdist-deps. Working
around this would require moving the fallback a couple of levels up the
stack to a component that knows about installing packages for the build
process. That's not impossible, but it's inelegant and less efficient.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 25 June 2017 at 17:26, Nathaniel Smith  wrote:
> On Sat, Jun 24, 2017 at 7:57 AM, Thomas Kluyver  wrote:
>> Nick has merged that PR, and the updated PEP is visible here:
>>
>> https://www.python.org/dev/peps/pep-0517/
>>
>> Hopefully we're nearing a consensus on this now. If you're interested,
>> please do have a read through the latest version.
>>
>
> Reading through it just now, there were some bits that gave me the
> impression that prepare_build_wheel_files must *always* be called
> before calling build_wheel (e.g. "Because the wheel will be built from
> a temporary build directory, build_wheel may create intermediate files
> in the working  directory, and does not need to take care to clean
> them up."). If we end up keeping prepare_build_wheel_files, then I
> think we should add some text making clear exactly which paths are
> legal.

Doing out-of-tree builds is completely optional for frontends, so the
possible build paths are:

In-place build:
* just call build_wheel on the source tree

Out-of-tree build (via sdist):
* call build_sdist
* unpack it into the build directory
* call build_wheel on the build directory

Out-of-tree build (prepare hook defined):
* call prepare_wheel_build_files
* call build_wheel on the prepared build directory

The critical constraint on front-ends is that they need to ensure the
appropriate dependencies are present before calling the affected hooks
- beyond that, they're entirely free to do things however they like
(e.g. always build in-place, always go via the sdist hook, or some
combination there-of).

Aside from not producing an archive, the bit that makes
`prepare_wheel_build_files` notably different from `build_sdist` is
that it relies on `get_build_wheel_requires` *not* on
`get_build_sdist_requires`:
https://www.python.org/dev/peps/pep-0517/#build-environment

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Sun, Jun 25, 2017 at 12:26 AM, Nick Coghlan  wrote:
> On 25 June 2017 at 16:58, Nathaniel Smith  wrote:
>> The current spec's solution:
>>
>> - Let's define an exhaustive list of all the reasons you might want to
>> make an sdist:
>>   1) you're a hypothetical future version of pip that has decided to
>> implement some specific build-from-unpacked-source-tree semantics
>>   2) the user has specifically requested an sdist, so if one can't be
>> made then the only thing to do is to fail and pass on some
>> unstructured error log
>>   3) that's it, we've made a list of all possible front-end semantics.
>> - Then define a hook that does (1) and a hook that does (2).
>
> pip doesn't care about making sdists, it only cares about doing out of
> tree wheel builds. It's other tools (e.g. tox) that specifically care
> about making sdists.
>
> So we only have two front-end scanarios that we need to handle:
>
> 1. preparing for an out-of-tree call to build_wheel
> 2. actually building an sdist (e.g. for testing or publication)

Right, the core point of disagreement is:

- I'm not convinced that this list is exhaustive.
- I'm not convinced that we understand the out-of-tree build case,
given that the sdist approach has never been implemented or deployed.
- Even based on our current imperfect understanding of the out-of-tree
build case, I'm not convinced that prepare_build_wheel_files solution
is actually the best solution. (For all the reasons I've said before:
the stated motivation for doing out-of-tree builds is that we don't
trust the build system, but prepare_build_wheel_files forces us to
trust the build system; flit *could* support sdists, the question is
whether it's worth the effort, and I don't believe we really
understand the trade-offs well enough to know the answer to that; in
the flit case copytree() would work just as well as
prepare_build_wheel_files anyway so there's no demonstrated need for
this complexity.)

Maybe you're right and there are exactly 2 front-end use cases and it
will turn out that the current PEP addresses them perfectly. I don't
have a crystal ball; I'm making an argument from ignorance. But I feel
like allowing NotImplemented returns + ext_{toolname}_* hooks seems
like it covers all the known cases about as well as the current
design, while being substantially simpler and more future-proof.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 25 June 2017 at 16:58, Nathaniel Smith  wrote:
> The current spec's solution:
>
> - Let's define an exhaustive list of all the reasons you might want to
> make an sdist:
>   1) you're a hypothetical future version of pip that has decided to
> implement some specific build-from-unpacked-source-tree semantics
>   2) the user has specifically requested an sdist, so if one can't be
> made then the only thing to do is to fail and pass on some
> unstructured error log
>   3) that's it, we've made a list of all possible front-end semantics.
> - Then define a hook that does (1) and a hook that does (2).

pip doesn't care about making sdists, it only cares about doing out of
tree wheel builds. It's other tools (e.g. tox) that specifically care
about making sdists.

So we only have two front-end scanarios that we need to handle:

1. preparing for an out-of-tree call to build_wheel
2. actually building an sdist (e.g. for testing or publication)

The possibility of only defining one hook arises from the fact that
providing just the build_sdist hook would be sufficient for both tasks
*if* we were willing to impose the constraint that everything a
backend depends on to build an sdist will always also be a
pre-requisite for building a wheel file.

Thomas has indicated that that *isn't* the case for flit - he expects
the `build_sdist` hook to need runtime dependencies (either Python
level or external) that `build_wheel` doesn't.

Thus `prepare_build_wheel_files` to cover the case where the frontend
doesn't really want an sdist at all, it just wants to do an
out-of-tree wheel build, but the backend doesn't want to ensure that
the preconditions for `build_sdist` and `build_wheel` are identical.

However, we've chosen to make the second hook optional, so backends
that have a very simple `build_sdist` implementation don't have to
worry about the optional wheel preparation hook.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Sat, Jun 24, 2017 at 7:57 AM, Thomas Kluyver  wrote:
> Nick has merged that PR, and the updated PEP is visible here:
>
> https://www.python.org/dev/peps/pep-0517/
>
> Hopefully we're nearing a consensus on this now. If you're interested,
> please do have a read through the latest version.
>

Reading through it just now, there were some bits that gave me the
impression that prepare_build_wheel_files must *always* be called
before calling build_wheel (e.g. "Because the wheel will be built from
a temporary build directory, build_wheel may create intermediate files
in the working  directory, and does not need to take care to clean
them up."). If we end up keeping prepare_build_wheel_files, then I
think we should add some text making clear exactly which paths are
legal.

...and in the current spec this is complex enough that something like
a state machine might be the clearest way? Here's an attempt at an NFA
representation.

Notation: [foo] is a state, --bar--> is a labeled transition.

Required transitions:

[VCS checkout] --build_wheel--> [wheel]

Required transitions if you want to have a useful sdist at all:

[VCS checkout] --build_sdist+unpack--> [non-VCS tree]
[non-VCS tree] --build_wheel--> [wheel]

Required if you have a prepare_wheel_metadata hook:

[VCS checkout] --prepare_wheel_metadata--> [checkout+metadata]
[checkout+metadata] --build_wheel(..., metadata_directory=...)--> [wheel]
[non-VCS tree] --prepare_wheel_metadata--> [non-VCS tree+metadata]
[non-VCS tree+metadata] --build_wheel(..., metadata_directory=...)--> [wheel]

Plus you should support EITHER:

[non-VCS tree] --build_sdist+unpack--> [non-VCS tree]

OR:

[VCS checkout] --prepare_wheel_build_files--> [non-VCS tree]
[non-VCS tree] --prepare_wheel_build_files--> [non-VCS tree]

...I'm not sure this is really making things clearer. (Though a
graphviz version would be less terrible than the text version, maybe?)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Sat, Jun 17, 2017 at 7:41 AM, Nick Coghlan  wrote:
> On 17 June 2017 at 17:07, Nathaniel Smith  wrote:
>> On Fri, Jun 16, 2017 at 11:28 PM, Donald Stufft  wrote:

 On Jun 16, 2017, at 11:08 PM, Nathaniel Smith  wrote:

 (because the legacy setup.py
 command errors out, because the build_sdist hook is missing, because
 the build_sdist hook errors out...); probably falling back on
 shutil.copytree.
>>>
>>>
>>> Ah okay. This is where the disconnect was. I assumed if a project was using 
>>> a PEP 517 build tool the result of a build_sdist failure (or a build_wheel 
>>> failure) would be surfacing an error to the end user, not some sort of 
>>> fallback.
>>
>> What we might be able to get away with in the PEP 517 transition is to
>> provide an explicit way of signalling "build_sdist failed because
>> that's just not a supported operation", so that pip could distinguish
>> that from "something unexpected blew up" and only fall back in the
>> first case.
>
> That's what `prepare_wheel_input_files` gives us: if the backend
> doesn't implement it, then `build_sdist` is expected to always
> succeed, and any errors should be reported to the end user.

Well, yeah, this is a discussion about a potentially simpler/more
flexible alternative to prepare_wheel_input_files, so I'd hope that it
gives us something similar :-).

> OTOH, if `build_sdist` can fail (e.g. due to missing dependencies that
> aren't needed just to build a wheel), then the backend should offer a
> `prepare_wheel_input_files` that offers the expected guarantee (i.e.
> it will only fail for reasons that should be reported to the user
> attempting to do the build).
>
> The frontend then *won't* have a fallback copy strategy for PEP 517
> backends - if `prepare_wheel_input_files` is present and fails, or if
> it's missing and `build_sdist` fails, then you have a failed wheel
> build on your hands.
>
> The legacy copying strategy would then only be used when falling all
> the way back to the `setup.py` interface, not as an attempt to
> continue the build in the face of a backend bug.
>
> This all gets murkier if you try to make the build_sdist hook more
> complex instead of just having two different hooks for frontends to
> call that tell the backend what the frontend actually wants (i.e.
> either an actual sdist, or just a clean set of exported files to use
> as input to a wheel build).

Hmm, I don't think it does get murkier. Here's how I'm thinking about it:

Everyone agrees:

- build_sdist's semantics are that it builds an sdist, it's up to the
caller to figure out why they want that
- the build_sdist hook can be missing (unless we're planning to
enforce its presence from the start, but that seems unlikely given
that pip currently has no sdist handling at all), which is a pretty
obvious signal to the caller that it's not supported
- there's some problem with what to do for flit, where "can I build an
sdist" is determined at runtime. I guess it could make the build_sdist
attribute be defined via a __getattr__ method that sniffs the source
tree before deciding whether the attribute should be present, but that
seems unpleasant.

The current spec's solution:

- Let's define an exhaustive list of all the reasons you might want to
make an sdist:
  1) you're a hypothetical future version of pip that has decided to
implement some specific build-from-unpacked-source-tree semantics
  2) the user has specifically requested an sdist, so if one can't be
made then the only thing to do is to fail and pass on some
unstructured error log
  3) that's it, we've made a list of all possible front-end semantics.
- Then define a hook that does (1) and a hook that does (2).

My suggestion:

- Let's keep the spec around build_sdist very simple, all it does is
worry about building an sdist and doesn't care at all what it's for,
that's the frontend's job
- Within that scope, one very well-defined thing that a backend might
want to say to a frontend is "this backend doesn't support building an
sdist from this source tree". And we even support this already, just
in a kinda awkward way requiring __getattr__.
- so instead let's provide a standard way to do that
- and then backends and the standard don't (yet) need to know anything
about pip's particular use case for sdists of wanting to potentially
in the future use them as an intermediate step in building wheels from
source directories; all the necessary logic is a concern for pip (and
possibly some future PEP), not for PEP 517.

(Hopefully this also answers Paul's objection – I'm explicitly trying
to take all the fallback-y logic out of the PEP, and replace it with a
nicely semantic "that operation isn't supported" signal. It just so
happens that one of the things pip could do with that signal is to
decide to use an alternative method of accomplishing its goal that
doesn't involve creating an sdist.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Dis

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

Nick has merged that PR, and the updated PEP is visible here:

https://www.python.org/dev/peps/pep-0517/

Hopefully we're nearing a consensus on this now. If you're interested,
please do have a read through the latest version.

Thomas

On Sat, Jun 24, 2017, at 02:44 PM, Thomas Kluyver wrote:
> I have prepared a PR against the PEP adding get_build_sdist_requires ,
> and renaming a couple of the other hooks for clarity
> (get_build_wheel_requires, prepare_build_wheel_files):
> 
> https://github.com/python/peps/pull/297
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

I have prepared a PR against the PEP adding get_build_sdist_requires ,
and renaming a couple of the other hooks for clarity
(get_build_wheel_requires, prepare_build_wheel_files):

https://github.com/python/peps/pull/297
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 17 June 2017 at 17:07, Nathaniel Smith  wrote:
> On Fri, Jun 16, 2017 at 11:28 PM, Donald Stufft  wrote:
>>>
>>> On Jun 16, 2017, at 11:08 PM, Nathaniel Smith  wrote:
>>>
>>> (because the legacy setup.py
>>> command errors out, because the build_sdist hook is missing, because
>>> the build_sdist hook errors out...); probably falling back on
>>> shutil.copytree.
>>
>>
>> Ah okay. This is where the disconnect was. I assumed if a project was using 
>> a PEP 517 build tool the result of a build_sdist failure (or a build_wheel 
>> failure) would be surfacing an error to the end user, not some sort of 
>> fallback.
>
> What we might be able to get away with in the PEP 517 transition is to
> provide an explicit way of signalling "build_sdist failed because
> that's just not a supported operation", so that pip could distinguish
> that from "something unexpected blew up" and only fall back in the
> first case.

That's what `prepare_wheel_input_files` gives us: if the backend
doesn't implement it, then `build_sdist` is expected to always
succeed, and any errors should be reported to the end user.

OTOH, if `build_sdist` can fail (e.g. due to missing dependencies that
aren't needed just to build a wheel), then the backend should offer a
`prepare_wheel_input_files` that offers the expected guarantee (i.e.
it will only fail for reasons that should be reported to the user
attempting to do the build).

The frontend then *won't* have a fallback copy strategy for PEP 517
backends - if `prepare_wheel_input_files` is present and fails, or if
it's missing and `build_sdist` fails, then you have a failed wheel
build on your hands.

The legacy copying strategy would then only be used when falling all
the way back to the `setup.py` interface, not as an attempt to
continue the build in the face of a backend bug.

This all gets murkier if you try to make the build_sdist hook more
complex instead of just having two different hooks for frontends to
call that tell the backend what the frontend actually wants (i.e.
either an actual sdist, or just a clean set of exported files to use
as input to a wheel build).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Paul Moore]

On 17 June 2017 at 08:07, Nathaniel Smith  wrote:
> Proposal: "NotImplemented" is a legal return value from build_sdist
> (similar to dunder methods), and should trigger whatever fallback
> behavior the frontend would do if the hook was simply undefined.

The PEP currently allows build_sdist to fail, but frontends don't have
much option do do anything other than present that failure direct to
the user. Not because there isn't a special "fall back to something
else" return value, but because the PEP provides no guarantee that any
fallback is possible. We've been very careful to *not* require
anything of source trees than what the hooks offer, as I understand
it, so it seems like we're doing a bit of a U-turn now if we say "if
the hook fails, front ends will fall back" and don't provide any
information as to what front ends are allowed to assume as part of
that fallback. You mention falling back to copying, but the PEP
doesn't even specify that source trees must be copyable. I'd probably
have to work quite hard to come up with a scenario where they aren't,
but if we want to make that guarantee, let's just be explicit about
it. A single sentence "backend operations on source trees must behave
the same if the source tree is copied to a different filesystem
location, and front ends are free to copy source trees if needed" is
easy to add, if that's what we want to guarantee.

Personally, I prefer not adding constraints on source trees, and
leaving the PEP as it is, with no fallback required or expected if a
hook fails.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Fri, Jun 16, 2017 at 11:28 PM, Donald Stufft  wrote:
>>
>> On Jun 16, 2017, at 11:08 PM, Nathaniel Smith  wrote:
>>
>> (because the legacy setup.py
>> command errors out, because the build_sdist hook is missing, because
>> the build_sdist hook errors out...); probably falling back on
>> shutil.copytree.
>
>
> Ah okay. This is where the disconnect was. I assumed if a project was using a 
> PEP 517 build tool the result of a build_sdist failure (or a build_wheel 
> failure) would be surfacing an error to the end user, not some sort of 
> fallback.

What we might be able to get away with in the PEP 517 transition is to
provide an explicit way of signalling "build_sdist failed because
that's just not a supported operation", so that pip could distinguish
that from "something unexpected blew up" and only fall back in the
first case.

Obviously we want build_sdist to be supported in as many places as
possible, but given that we anticipate that there will be times when
it won't be (flit, or ad hoc build systems that just never implement
it) then I think having a standard way to communicate that is a good
idea regardless.

It's surprisingly tricky to come up with a good signal for this
though. We don't want to use a user-defined exception, because we
don't have like a "standard PEP 517 library" to put it in. We could
use a built-in exception like NotImplementedError, but then there's
the risk that that gets raised for some other reason  internally, it
leaks out, and then we misinterpret it as being an intentional signal.

Proposal: "NotImplemented" is a legal return value from build_sdist
(similar to dunder methods), and should trigger whatever fallback
behavior the frontend would do if the hook was simply undefined.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Donald Stufft]

> 
> On Jun 16, 2017, at 11:08 PM, Nathaniel Smith  wrote:
> 
> (because the legacy setup.py
> command errors out, because the build_sdist hook is missing, because
> the build_sdist hook errors out...); probably falling back on
> shutil.copytree.


Ah okay. This is where the disconnect was. I assumed if a project was using a 
PEP 517 build tool the result of a build_sdist failure (or a build_wheel 
failure) would be surfacing an error to the end user, not some sort of 
fallback. 
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Fri, Jun 16, 2017 at 7:48 PM, Nick Coghlan  wrote:
> On 17 June 2017 at 07:48, Nathaniel Smith  wrote:
>> Hmm, here's another plea for simplicity, but from a slightly different
>> direction that I just thought of: what if we said that any hooks
>> starting with "ext_pip_..." are reserved for pip's use, and pip can
>> make up whatever semantics it likes for them. And then as the parts of
>> pip that actually want to use prepare_wheel_metadata and/or
>> get_prepared_wheel_input_files come online, we use the ext_pip_*
>> versions of those hooks to prototype them and work out any issues. And
>> then once there's an actual implementation and proven value, we write
>> a new PEP to drop the ext_pip_ prefix and make them standard.
>>
>> What do you think?
>
> pip's needs here aren't unique to pip - they're just "system
> integrator" needs, rather than "software developer" needs.
>
> It's fairly common for system integrator needs to seem like pointless
> complexity to folks more focused on addressing the "single component
> publisher (aka software developer)" case, but that's part of why we
> have the PEP process: to encourage us to figure out the best place for
> complexity to live in order to advance the overall Python ecosystem,
> rather than trying to convince ourselves that the underlying
> complexity doesn't exist.

I'm not making an argument that the complexity doesn't exist or that
pip is the only project that matters; I'm arguing that it's risky to
standardize things before we've really hit the problem, because we
might not fully understand it yet. Which is a truism :-). But
sometimes you don't have much choice. The real suggestion is hey,
maybe in this case there's a way to get these
important-but-not-fully-understood issues off the critical path for
PEP 517 without losing our chance to handle them later.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Fri, Jun 16, 2017 at 5:52 PM, Donald Stufft  wrote:
> I’d probably want to spec this out as ext_{name-on-pypi}_* to remove a
> special case on pip in the PEP, to let others do experimentation as well.

Sure, like the [tool.*] escape hatch in pyproject.toml.

> However this solution is fine with me on both of the Non build_{sdist,wheel}
> hooks. Although I feel like it kind of defeats the purpose of the
> prepare_build_files hooks, because I think Thomas’ goal is to not have to
> support build_sdist from within another sdist, and if he depends on a pip
> extension (rather than it being an optional speed up like
> prepare_wheel_metadata) then we either break flit users if we decide it
> wasn’t worth it and we remove it, or we force other frontends to understand
> ext_pip_prepare_build_files and we get into the same kind of “everyone is
> just trying to emulate setuptools” bog that we’re trying to escape.

How I'm imagining it would work is:

Right now: pip just unconditionally uses shutil.copytree, and doesn't
even try to generate an sdist.

Eventually: someone writes the patch to pip to attempt to do the copy
step via generating an sdist. This is going to require some fallback
strategy for when building an sdist fails (because the legacy setup.py
command errors out, because the build_sdist hook is missing, because
the build_sdist hook errors out...); probably falling back on
shutil.copytree. For flit's purposes, this doesn't seem like a
disaster: flit projects don't have tens of megabytes of random build
artifacts lying around, and flit sdists in particular don't have tens
of megabytes of irrelevant VCS history. In fact, an unpacked flit
sdist is generally just as cheap to copy via shutil.copytree as any
other project would be to copy via build_sdist.

Eventually^2 (possibly part of the previous step): someone comes up
with a prototype ext_pip_prepare_build_files spec to optimize this
further, and it gets implemented in pip and flit. Experience shows
that the speedup is worthwhile, enough so that other build frontends
want to get in on the action.

Eventually^3: we write up a PEP that's just a copy/paste of the mature
pip behavior and tell other frontends to go for it.

But of course there are also other ways this could go. Like, I'm not
saying it's likely, but for all we know, between now and when pip gets
around to implementing the copy-via-sdist approach, Thomas might have
some brainstorm and realize that flit needs to implement
build-sdist-from-sdist for some other reason. Making predictions is
hard, especially about the future.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 17 June 2017 at 07:48, Nathaniel Smith  wrote:
> Hmm, here's another plea for simplicity, but from a slightly different
> direction that I just thought of: what if we said that any hooks
> starting with "ext_pip_..." are reserved for pip's use, and pip can
> make up whatever semantics it likes for them. And then as the parts of
> pip that actually want to use prepare_wheel_metadata and/or
> get_prepared_wheel_input_files come online, we use the ext_pip_*
> versions of those hooks to prototype them and work out any issues. And
> then once there's an actual implementation and proven value, we write
> a new PEP to drop the ext_pip_ prefix and make them standard.
>
> What do you think?

pip's needs here aren't unique to pip - they're just "system
integrator" needs, rather than "software developer" needs.

It's fairly common for system integrator needs to seem like pointless
complexity to folks more focused on addressing the "single component
publisher (aka software developer)" case, but that's part of why we
have the PEP process: to encourage us to figure out the best place for
complexity to live in order to advance the overall Python ecosystem,
rather than trying to convince ourselves that the underlying
complexity doesn't exist.

The funding dynamics of open source *do* play into those discussions
as a design consideration, since any work done by volunteers needs to
offer some kind of inherent benefit to those volunteers (or they won't
have any incentive to do the work), while work that we expect to be
done by professionals on work time requires some form of justification
as being in their customer's interests (e.g. "more software available
more reliably in the formats that you prefer").

Cheers,
Nick.

P.S. This "Software distribution, even over the internet, is a more
complex problem than it may first appear" is also why PEP 426 opens
with an attempted overview of the full complexity of the "original
software component publisher to downstream application, service, or
appliance user", and why that's also one of the points I dwelled on in
[1]. Large scale systems engineering is a distinct engineering
discipline for a reason, and it mostly boils down to making smart
decisions about how you divide responsibilities between components.

[1] 
http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

On 17 June 2017 at 08:05, Thomas Kluyver  wrote:
> On Fri, Jun 16, 2017, at 10:48 PM, Nathaniel Smith wrote:
>> The messy complications come from
>> prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't
>> surprising, since those are the two hooks where we're squinting into
>> our crystal ball to try and guess what will be useful for software
>> that doesn't exist yet, but will later, maybe, we hope.
>
> I'm not exactly clear on what use cases the prepare_wheel_metadata hook
> satisfies - that was in the spec before I was particularly involved in
> it.

As Donald notes, this is primarily a workaround for the
underspecification of the sdist format - wheel metadata is the most
trustworthy information source we have for post-installation project
metadata (name, version, runtime dependencies, etc), since it's
exactly what will end up in the post-install dist-info directory
(minus the files generated by an installer at installation time).

Pure-Python backends like flit can probably skip providing it, since
building a wheel is a cheap operation, but something like enscons is
probably going to want to implement it (since only generating the
Python wheel metadata should be much cheaper than running the full
build).

And unlike defining a new iteration of the sdist format, encouraging
backend developers to make wheel metadata generation a cheap operation
doesn't require any new speculative format definition work on our
part.

> I do think we've hashed out a concrete need for prepare_build_files (or
> whatever it ends up called): to copy the files to a build directory
> without either copying large amounts of unnecessary data or figuring out
> what belongs in an sdist. The only alternative would be to require
> backends to avoid leaving clutter in the working directory, so they
> would take care of any necessary copy step rather than exposing a
> separate hook. People insist that trusting backends is a non-starter,
> though.

It isn't that trusting them is a non-starter in all situations, it's
that *requiring* front-ends to trust them is a non-starter.

Instead we want front-ends to have the following expectation:

1. either a backend's build_sdist is always fast, and doesn't require
any dependencies not already needed to build a wheel;
2. or else, if build_sdist isn't always fast, or requires extra
dependencies, then prepare_wheel_input_files will be provided

"Dependencies" there refers to both actual Python-level dependencies
expressible in terms of PEP 508, as well as external dependencies that
currently can't be represented (e.g. requiring particular command line
tools to be available).

Cheers,
Nick.

P.S. This also seems like an opportune time to remind folks that
Tennessee Leeuwenburg and Robert Collins started a draft PEP for
expressing external dependencies a couple of years ago:
https://github.com/pypa/interoperability-peps/pull/30/files

While any further work along those lines would need significant
updates to account for PEP 508, PEP 518, PEP 517, and other
developments, the core concept of using
"" so that external
dependencies can potentially be passed anywhere that PEP 508
dependencies are currently supported remains sound, and would likely
provide a solid foundation for a plugin based model where (for
example), there might be common "bin", "clib", and "cheader" types at
the dependency declaration level (but platform dependent
implementations of those plugins), as well as cross-platform
namespaces for 3rd party dependency management ecosystems (e.g. npm,
cargo, maven) (which would then either have appropriate build system
requirements like "bin!!npm", "bin!!cargo" and "bin!!maven",
dependencies on Python wrappers for those tools, or else dependencies
on suitable extras in the underlying build backend).

The one place where we likely *wouldn't* mix internal and external
requirements is in interoperability specifications: there, we'd keep
the external requirements separate, so that older tools don't get
confused trying to parse them, and instead just treat satisfying them
as somebody else's problem (i.e. preserving the status quo).

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Donald Stufft]

> On Jun 16, 2017, at 5:48 PM, Nathaniel Smith  wrote:
> 
> On Fri, Jun 16, 2017 at 2:08 AM, Thomas Kluyver  > wrote:
>> 
>> On Fri, Jun 16, 2017, at 08:41 AM, Nick Coghlan wrote:
 I think we should rename get_build_requires to
 get_build_wheel_requires, and add a get_build_sdist_requires. And the
 rule would be:
 
 get_build_sdist_requires: can assume build-system.requires are available
 get_build_wheel_requires: can assume build-system.requires are available
 
 build_sdist: can assume build-system.requires and
 get_build_sdist_requires are available
 prepare_wheel_metadata, build_wheel: can assume build-system.requires
 and get_build_wheel_requires are available
>>> 
>>> +1 from me
>> 
>> *Sigh*, another hook. It makes sense in context, but I can't shake the
>> feeling that what was a relatively simple spec is steadily turning into
>> a complex monster. I still resent that we're trying to standardise an
>> interface to build sdists at the same time as one to build wheels.
> 
> Hmm, here's another plea for simplicity, but from a slightly different
> direction that I just thought of: what if we said that any hooks
> starting with "ext_pip_..." are reserved for pip's use, and pip can
> make up whatever semantics it likes for them. And then as the parts of
> pip that actually want to use prepare_wheel_metadata and/or
> get_prepared_wheel_input_files come online, we use the ext_pip_*
> versions of those hooks to prototype them and work out any issues. And
> then once there's an actual implementation and proven value, we write
> a new PEP to drop the ext_pip_ prefix and make them standard.
> 


I’d probably want to spec this out as ext_{name-on-pypi}_* to remove a special 
case on pip in the PEP, to let others do experimentation as well. However this 
solution is fine with me on both of the Non build_{sdist,wheel} hooks. Although 
I feel like it kind of defeats the purpose of the prepare_build_files hooks, 
because I think Thomas’ goal is to not have to support build_sdist from within 
another sdist, and if he depends on a pip extension (rather than it being an 
optional speed up like prepare_wheel_metadata) then we either break flit users 
if we decide it wasn’t worth it and we remove it, or we force other frontends 
to understand ext_pip_prepare_build_files and we get into the same kind of 
“everyone is just trying to emulate setuptools” bog that we’re trying to escape.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Donald Stufft]

> On Jun 16, 2017, at 6:05 PM, Thomas Kluyver  wrote:
> 
> On Fri, Jun 16, 2017, at 10:48 PM, Nathaniel Smith wrote:
>> The messy complications come from
>> prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't
>> surprising, since those are the two hooks where we're squinting into
>> our crystal ball to try and guess what will be useful for software
>> that doesn't exist yet, but will later, maybe, we hope.
> 
> I'm not exactly clear on what use cases the prepare_wheel_metadata hook
> satisfies - that was in the spec before I was particularly involved in
> it.


Basically it exists because when we’re resolving dependencies to decide what 
set of projects/versions to install, we need a way to take a sdist (because a 
wheel might not exist!) and decide what dependencies it needs. We ultimately 
might end up throwing this away because maybe it ends up conflicting and we 
can’t actually utilize this particular version at all. So we might end up 
cycling through 10 different versions of the same project looking for a version 
whose dependencies don’t conflict with the rest of what we’re trying to install.

Without prepare_wheel_metadata our only real option is to just build the wheel 
and inspect that. For a tool like flit that is probably fine, because since 
it’s pure Python only building a wheel is going to be exceedingly quick. 
However something like Scikit or something where building the wheel might take 
tens of minutes or longer that can degrade things very quickly. If you imagine 
something that takes even 5 minutes to go from sdist to wheel, if we need to do 
that 10 times while looking for a version that we can use, that means 
resolution ends up taking over an hour just because of that package alone.

This cost is tempered somewhat by the fact that in the ideal case we can cache 
those wheels so in the future resolution can be extremely quick, however not 
everyone can or will run with caching enabled and even then, ``pip install …`` 
taking an hour even just the first time is still a pretty horrible user 
experience.

The goal of prepare_wheel_metadata then is basically allowing us to ask a sdist 
“what dependencies would you have, if we were to build you as a wheel” without 
having to go through the entire build process. This will ideally be much much 
faster.

It’s not *just* dependencies either, in many cases we don’t know the name or 
version of something because we’re just given a fairly generic tarball (for 
instance `master.zip` from Github). We need a way to get name/version from that 
tarball, which would exist in the wheel but we end up falling into the same 
“but what if things take forever to build” problem.

Another possible solution is to go down the path of trying to make a sdist 2.0 
that has this metadata in a static format so we can promise that we can get it 
without needing to build anything. However not only is that a significant chunk 
of extra work, but some folks (Nathaniel I think?) has indicated that some 
projects simply can’t determine their dependencies statically at sdist build 
time, because it’s going to change at build time (I think the example was 
something can build against Numpy >=X, but once built against X+5, it has to 
have Numpy >=X.5 at runtime).


> 
> I do think we've hashed out a concrete need for prepare_build_files (or
> whatever it ends up called): to copy the files to a build directory
> without either copying large amounts of unnecessary data or figuring out
> what belongs in an sdist. The only alternative would be to require
> backends to avoid leaving clutter in the working directory, so they
> would take care of any necessary copy step rather than exposing a
> separate hook. People insist that trusting backends is a non-starter, 
> though.


The other solution of course is to just say that all backends needs to be able 
to no-op copy a sdist from an existing sdist. So we have three options really:

1) Require backends to be able to no-op copy a sdist from an existing unpacked 
sdist. However Thomas is against this, as it would make flit’s job harder.
2) Require frontends to trust that backends are going to DTRT with regards to 
in-place builds and isolation. However myself and the other pip devs were 
against this, as we feel it is important for pip to do it’s job.
3) Add a hook that let’s the backends copy the files it needs into a staging 
area, without the need to prepare a full sdist. This is basically (1) except 
with some compromises to make the use cases that Thomas says makes flit’s job 
harder easier to deal with. 

Nathaniel is against (3) for simplicity sake, and personally I would prefer (1) 
because I think it is simpler and because I think that it shouldn’t be *too* 
much additional effort for a backend to make a no-op build_sdist in the case 
they’re being made out of something that is already a sdist. That being said, I 
am OK with (3) since Thomas believes that is better for flit than (1) and I 
don’t have any e

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

On Fri, Jun 16, 2017, at 10:48 PM, Nathaniel Smith wrote:
> The messy complications come from
> prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't
> surprising, since those are the two hooks where we're squinting into
> our crystal ball to try and guess what will be useful for software
> that doesn't exist yet, but will later, maybe, we hope.

I'm not exactly clear on what use cases the prepare_wheel_metadata hook
satisfies - that was in the spec before I was particularly involved in
it.

I do think we've hashed out a concrete need for prepare_build_files (or
whatever it ends up called): to copy the files to a build directory
without either copying large amounts of unnecessary data or figuring out
what belongs in an sdist. The only alternative would be to require
backends to avoid leaving clutter in the working directory, so they
would take care of any necessary copy step rather than exposing a
separate hook. People insist that trusting backends is a non-starter, 
though.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Fri, Jun 16, 2017 at 2:08 AM, Thomas Kluyver  wrote:
>
> On Fri, Jun 16, 2017, at 08:41 AM, Nick Coghlan wrote:
> > > I think we should rename get_build_requires to
> > > get_build_wheel_requires, and add a get_build_sdist_requires. And the
> > > rule would be:
> > >
> > > get_build_sdist_requires: can assume build-system.requires are available
> > > get_build_wheel_requires: can assume build-system.requires are available
> > >
> > > build_sdist: can assume build-system.requires and
> > > get_build_sdist_requires are available
> > > prepare_wheel_metadata, build_wheel: can assume build-system.requires
> > > and get_build_wheel_requires are available
> >
> > +1 from me
>
> *Sigh*, another hook. It makes sense in context, but I can't shake the
> feeling that what was a relatively simple spec is steadily turning into
> a complex monster. I still resent that we're trying to standardise an
> interface to build sdists at the same time as one to build wheels.

Yeah. Well, except I'm not *too* bothered by the sdist part in
particular, since the spec already had to define what an sdist was.
And if you zoom out then we have get_build_{wheel,sdist}_requires
which have identical interfaces, and build_{wheel,sdist} which also
have identical interfaces, so the total complexity for this part isn't
too high at all. The messy complications come from
prepare_wheel_metadata and get_prepare_wheel_input_files, which isn't
surprising, since those are the two hooks where we're squinting into
our crystal ball to try and guess what will be useful for software
that doesn't exist yet, but will later, maybe, we hope.

Hmm, here's another plea for simplicity, but from a slightly different
direction that I just thought of: what if we said that any hooks
starting with "ext_pip_..." are reserved for pip's use, and pip can
make up whatever semantics it likes for them. And then as the parts of
pip that actually want to use prepare_wheel_metadata and/or
get_prepared_wheel_input_files come online, we use the ext_pip_*
versions of those hooks to prototype them and work out any issues. And
then once there's an actual implementation and proven value, we write
a new PEP to drop the ext_pip_ prefix and make them standard.

What do you think?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Thomas Kluyver]

On Fri, Jun 16, 2017, at 08:41 AM, Nick Coghlan wrote:
> > I think we should rename get_build_requires to
> > get_build_wheel_requires, and add a get_build_sdist_requires. And the
> > rule would be:
> >
> > get_build_sdist_requires: can assume build-system.requires are available
> > get_build_wheel_requires: can assume build-system.requires are available
> >
> > build_sdist: can assume build-system.requires and
> > get_build_sdist_requires are available
> > prepare_wheel_metadata, build_wheel: can assume build-system.requires
> > and get_build_wheel_requires are available
> 
> +1 from me

*Sigh*, another hook. It makes sense in context, but I can't shake the
feeling that what was a relatively simple spec is steadily turning into
a complex monster. I still resent that we're trying to standardise an
interface to build sdists at the same time as one to build wheels.

> Another example: Thomas expects flit to require VCS interaction
> support for sdist generation, but not for wheel building (or wheel
> build file preparation).

It's not much help for this, though, because I can't specify git as a
dependency.

> My rationale for requiring get_build_wheel_requires to be called in
> the source directory is that it means that `prepare_wheel_input_files`
> can rely on those dynamic dependencies, appropriately reflecting it's
> status as an optional substep of the `build wheel` process.

That makes sense to me.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nick Coghlan]

Thanks for continuing to push us forward on this, Thomas :)

A small PEP readability request: given how the number of hooks has
grown, could we get a section that just lists the required hooks and
the optional hooks?

Alternatively, give each hook its own subsection under "Build backend
interface", so the table of contents at the top of the PEP serves as a
quick summary.

On 16 June 2017 at 06:43, Nathaniel Smith  wrote:
> On Thu, Jun 15, 2017 at 6:12 AM, Thomas Kluyver  wrote:
>> Yes, it's PEP 517 again! Here's the current text:
>>
>> https://www.python.org/dev/peps/pep-0517/
>>
>> We currently say:
>>
>>> All other hooks [except get_build_requires] are executed in an environment 
>>> which contains both the bootstrap requirements specified in the 
>>> pyproject.toml hook and those specified by the get_build_requires hook.
>>
>> It's not clear to me whether this should be required for the build_sdist
>> and prepare_build_files hooks, nor whether there are any adverse
>> consequences of specifying it like this anyway. Thoughts?
>
> I think we should rename get_build_requires to
> get_build_wheel_requires, and add a get_build_sdist_requires. And the
> rule would be:
>
> get_build_sdist_requires: can assume build-system.requires are available
> get_build_wheel_requires: can assume build-system.requires are available
>
> build_sdist: can assume build-system.requires and
> get_build_sdist_requires are available
> prepare_wheel_metadata, build_wheel: can assume build-system.requires
> and get_build_wheel_requires are available

+1 from me

> Rationale: (a) conceptually the sdist and wheel phases are totally
> separate, so we shouldn't couple them by requiring a single hook to
> provide the union of requirements for both. (b) there are known cases
> where building an sdist has different requirements than building a
> wheel. Examples: packages that run cython at sdist generation time.

Another example: Thomas expects flit to require VCS interaction
support for sdist generation, but not for wheel building (or wheel
build file preparation).

> It's not immediately obvious to me how prepare_build_files would fit
> into this; if it's only supposed to be used for building from an
> sdist, then it's like an extra half-phase in between the sdist and
> wheel phases -- maybe it's more part of the wheel phase and should get
> the wheel requirements? Is that the only time it's used? I guess I'll
> wait to worry about it until after I see how people respond to my
> argument in the other thread that prepare_build_files shouldn't exist
> at all :-).

As with get_build_requires, this was less ambiguous when "build_wheel"
was the only build hook. Now that we have "build_sdist" as well, then
it may make more sense to rename it to "prepare_wheel_input_files".

Something else that I believe the PEP currently leaves implicit is the
assumptions that backend hook implementations are allowed to make
regarding the current working directory when they're invoked.

As I understand it, there are three defined possibilities:

- original source tree (potentially VCS metadata, no PKG-INFO file)
- unpacked sdist (no VCS metadata, PKG-INFO file)
- prepared wheel input files (no VCS metadata, maybe PKG-INFO file)

Given those options, the hooks that can be called given a particular
kind of input directory are:

* original source tree:
* all hooks
* unpacked sdist:
* all hooks
* prepared wheel input files (if `prepare_wheel_input_files` is defined):
* prepare_wheel_metadata
* build_wheel
* NOT get_build_wheel_requires (see below)

My rationale for requiring get_build_wheel_requires to be called in
the source directory is that it means that `prepare_wheel_input_files`
can rely on those dynamic dependencies, appropriately reflecting it's
status as an optional substep of the `build wheel` process.

The available dependencies for each hook are then:

* build-system.requires only:
* get_build_sdist_requires
* get_build_wheel_requires

* build-system.requires + get_build_sdist_requires:
* build_sdist

* build-system.requires + get_build_wheel_requires:
* prepare_wheel_input_files
* prepare_wheel_metadata
* build_wheel

Make sense?

If folks agree with that, we could make the above explicit in the PEP
using a pair of dedicated paragraphs in each hook description:

* One starting "Current working directory: ..."
* One starting "Available dependencies: ..."

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Finishing up PEP 517

[Nathaniel Smith]

On Thu, Jun 15, 2017 at 6:12 AM, Thomas Kluyver  wrote:
> Yes, it's PEP 517 again! Here's the current text:
>
> https://www.python.org/dev/peps/pep-0517/
>
> We currently say:
>
>> All other hooks [except get_build_requires] are executed in an environment 
>> which contains both the bootstrap requirements specified in the 
>> pyproject.toml hook and those specified by the get_build_requires hook.
>
> It's not clear to me whether this should be required for the build_sdist
> and prepare_build_files hooks, nor whether there are any adverse
> consequences of specifying it like this anyway. Thoughts?

I think we should rename get_build_requires to
get_build_wheel_requires, and add a get_build_sdist_requires. And the
rule would be:

get_build_sdist_requires: can assume build-system.requires are available
get_build_wheel_requires: can assume build-system.requires are available

build_sdist: can assume build-system.requires and
get_build_sdist_requires are available
prepare_wheel_metadata, build_wheel: can assume build-system.requires
and get_build_wheel_requires are available

Rationale: (a) conceptually the sdist and wheel phases are totally
separate, so we shouldn't couple them by requiring a single hook to
provide the union of requirements for both. (b) there are known cases
where building an sdist has different requirements than building a
wheel. Examples: packages that run cython at sdist generation time.
(This is not *as* necessary if we have robust build-time requirement
support, but that's only one of the motivations for projects to do
this now; others include avoiding a potentially expensive step for
users -- installing cython in slow -- and avoiding the risk of later
cython changes breaking an existing release.) In the broader world,
it's very common to run automake/autotools etc. to generate
./configure scripts at sdist generation time. Maybe a python web app
needs node installed to run its asset pipeline so they do that at
sdist generation time rather than forcing end-users to install node.
Etc.

It's not immediately obvious to me how prepare_build_files would fit
into this; if it's only supposed to be used for building from an
sdist, then it's like an extra half-phase in between the sdist and
wheel phases -- maybe it's more part of the wheel phase and should get
the wheel requirements? Is that the only time it's used? I guess I'll
wait to worry about it until after I see how people respond to my
argument in the other thread that prepare_build_files shouldn't exist
at all :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Finishing up PEP 517

[Thomas Kluyver]

Yes, it's PEP 517 again! Here's the current text:

https://www.python.org/dev/peps/pep-0517/

We currently say:

> All other hooks [except get_build_requires] are executed in an environment 
> which contains both the bootstrap requirements specified in the 
> pyproject.toml hook and those specified by the get_build_requires hook.

It's not clear to me whether this should be required for the build_sdist
and prepare_build_files hooks, nor whether there are any adverse
consequences of specifying it like this anyway. Thoughts?

I'm going to start putting together a wrapper to call the PEP517 hooks
in a subprocess, which could be used by frontends.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig