Re: [Distutils] PEP 517 again

[Chris Barker]

On Mon, Sep 4, 2017 at 6:00 AM, Nick Coghlan  wrote:

> >> Do the Linux distros use pip to build their packages?
> >
> > Not that I am aware of.
>
> Fedora's build macros for Python projects currently rely on running
> setup.py directly, but we've been considering switching to pip instead
> since 2013 or so. PEP 517 is likely to provide the impetus to switch
> from "maybe we should do that" to "we need to do that, at least if
> setup.py is missing, and potentially always, so we get more consistent
> installation metadata"
>

which is exactly why I tried to do it in conda. In a post PEP 517 world,
that may be a good way to go. I'm looking forward to it.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Supposedly there is some meeting tomorrow concerning the wheel project that
will determine the fate of dist_info. So that is why I bought it up.

On Sep 4, 2017 9:00 PM, "Chris Jerdonek"  wrote:

> On Mon, Sep 4, 2017 at 6:41 PM, xoviat  wrote:
> > The PR that I am taking about is not for pip but for the wheel project.
>
> Okay, well you started the thread asking something similar for your
> pip PR (see below). I'm sure similar considerations hold for the wheel
> project.
>
> On Mon, Sep 4, 2017 at 5:11 PM, xoviat  wrote:
> > Also if someone with pip write access could please discuss and hopefully
> > merge my initial PR on pip, I would very much appreciate it. Paul seems
> to
> > be short on time.
>
> --Chris
>
>
> >
> > On Sep 4, 2017 8:19 PM, "Chris Jerdonek" 
> wrote:
> >>
> >> On Mon, Sep 4, 2017 at 6:08 PM, xoviat  wrote:
> >> > In any case, we're going to need this for prepare_metadata, so the
> >> > question
> >> > you should ask is: what are the reasons for *not* merging this? I
> >> > haven't
> >> > heard any so far but that doesn't mean that they don't exist. If there
> >> > are
> >> > none, then I don't see why we cannot merge my wheel PR and do a
> release.
> >>
> >> FYI, you're not the only one waiting for PR's to be merged. There are
> >> 56 other PR's, some of which have already been approved by repo
> >> members with commit access. I would try to be a little more patient.
> >> Otherwise, everyone can be emailing the list saying the same thing and
> >> asking why their PR isn't being merged.
> >>
> >> --Chris
> >>
> >> >
> >> > 2017-09-04 19:51 GMT-05:00 xoviat :
> >> >>
> >> >> > The only reason I can think of that setuptools would need a
> dist_info
> >> >> command would be to implement the PEP 517 prepare_wheel_metadata
> hook.
> >> >>
> >> >> Yes. That is absolutely correct.
> >> >>
> >> >> > But this hook is optional and in fact provides no value right now,
> so
> >> >> it can't be a blocker for anything.
> >> >>
> >> >> The simplest way to start on this issue is to replace egg_info in pip
> >> >> with
> >> >> prepare_metadata_for_build_wheel. It is absolutely a historical
> >> >> artifact but
> >> >> I need to work on one issue at a time. The next issue will be
> replacing
> >> >> egg_info in pip with prepare_metadata_for_build_wheel.
> >> >>
> >> >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith :
> >> >>>
> >> >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat  wrote:
> >> >>> > Nathaniel:
> >> >>> >
> >> >>> > Pip requires egg_info to discover dependencies of source
> >> >>> > distributions
> >> >>> > so
> >> >>> > that it can build wheels all at once after downloading the
> >> >>> > requirements. I
> >> >>> > need to move pip off of egg_info as soon as possible and dist_info
> >> >>> > is
> >> >>> > required to do that.
> >> >>>
> >> >>> "Requires" is a strong word -- AFAIK this is just a historical
> >> >>> artifact. I don't really know what you're talking about in the
> second
> >> >>> sentence.
> >> >>>
> >> >>> The only reason I can think of that setuptools would need a
> dist_info
> >> >>> command would be to implement the PEP 517 prepare_wheel_metadata
> hook.
> >> >>> But this hook is optional and in fact provides no value right now,
> so
> >> >>> it can't be a blocker for anything. (In fact I can't see any reason
> >> >>> why pip would ever call it before the resolver lands.) So either (a)
> >> >>> there's some other reason you want a dist_info command, (b) there's
> >> >>> some reason I'm missing why prepare_wheel_metadata matters, or (c)
> one
> >> >>> of us is misunderstanding something :-).
> >> >>>
> >> >>> -n
> >> >>>
> >> >>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
> >> >>> >>
> >> >>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat 
> wrote:
> >> >>> >> > Just an update for everyone here:
> >> >>> >> >
> >> >>> >> > 1. We're currently waiting on the implementation of the
> >> >>> >> > 'dist_info"
> >> >>> >> > command
> >> >>> >> > in the wheel project.
> >> >>> >> > 2. Once that is done we can switch pip over to reading
> dist-info
> >> >>> >> > rather
> >> >>> >> > than
> >> >>> >> > egg_info.
> >> >>> >> > 3. Then we can move the backend over to setuptools. Because
> Jacob
> >> >>> >> > has a
> >> >>> >> > much
> >> >>> >> > more efficient release system than pip, I anticipate having a
> >> >>> >> > release of
> >> >>> >> > setuptools first and then we can switch pip over to requiring a
> >> >>> >> > newer
> >> >>> >> > setuptools via PEP 518.
> >> >>> >>
> >> >>> >> I don't think pip actually has any use for the PEP 517
> >> >>> >> prepare_wheel_metadata hook right now though? Historically
> >> >>> >> 'setup.py
> >> >>> >> egg-info' was needed to kluge around unwanted behavior in
> 'setup.py
> >> >>> >> install', but with a PEP 517 backend that's irrelevant because
> >> >>> >> 

Re: [Distutils] PEP 517 again

[Chris Jerdonek]

On Mon, Sep 4, 2017 at 6:41 PM, xoviat  wrote:
> The PR that I am taking about is not for pip but for the wheel project.

Okay, well you started the thread asking something similar for your
pip PR (see below). I'm sure similar considerations hold for the wheel
project.

On Mon, Sep 4, 2017 at 5:11 PM, xoviat  wrote:
> Also if someone with pip write access could please discuss and hopefully
> merge my initial PR on pip, I would very much appreciate it. Paul seems to
> be short on time.

--Chris


>
> On Sep 4, 2017 8:19 PM, "Chris Jerdonek"  wrote:
>>
>> On Mon, Sep 4, 2017 at 6:08 PM, xoviat  wrote:
>> > In any case, we're going to need this for prepare_metadata, so the
>> > question
>> > you should ask is: what are the reasons for *not* merging this? I
>> > haven't
>> > heard any so far but that doesn't mean that they don't exist. If there
>> > are
>> > none, then I don't see why we cannot merge my wheel PR and do a release.
>>
>> FYI, you're not the only one waiting for PR's to be merged. There are
>> 56 other PR's, some of which have already been approved by repo
>> members with commit access. I would try to be a little more patient.
>> Otherwise, everyone can be emailing the list saying the same thing and
>> asking why their PR isn't being merged.
>>
>> --Chris
>>
>> >
>> > 2017-09-04 19:51 GMT-05:00 xoviat :
>> >>
>> >> > The only reason I can think of that setuptools would need a dist_info
>> >> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>> >>
>> >> Yes. That is absolutely correct.
>> >>
>> >> > But this hook is optional and in fact provides no value right now, so
>> >> it can't be a blocker for anything.
>> >>
>> >> The simplest way to start on this issue is to replace egg_info in pip
>> >> with
>> >> prepare_metadata_for_build_wheel. It is absolutely a historical
>> >> artifact but
>> >> I need to work on one issue at a time. The next issue will be replacing
>> >> egg_info in pip with prepare_metadata_for_build_wheel.
>> >>
>> >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith :
>> >>>
>> >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat  wrote:
>> >>> > Nathaniel:
>> >>> >
>> >>> > Pip requires egg_info to discover dependencies of source
>> >>> > distributions
>> >>> > so
>> >>> > that it can build wheels all at once after downloading the
>> >>> > requirements. I
>> >>> > need to move pip off of egg_info as soon as possible and dist_info
>> >>> > is
>> >>> > required to do that.
>> >>>
>> >>> "Requires" is a strong word -- AFAIK this is just a historical
>> >>> artifact. I don't really know what you're talking about in the second
>> >>> sentence.
>> >>>
>> >>> The only reason I can think of that setuptools would need a dist_info
>> >>> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>> >>> But this hook is optional and in fact provides no value right now, so
>> >>> it can't be a blocker for anything. (In fact I can't see any reason
>> >>> why pip would ever call it before the resolver lands.) So either (a)
>> >>> there's some other reason you want a dist_info command, (b) there's
>> >>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
>> >>> of us is misunderstanding something :-).
>> >>>
>> >>> -n
>> >>>
>> >>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
>> >>> >>
>> >>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
>> >>> >> > Just an update for everyone here:
>> >>> >> >
>> >>> >> > 1. We're currently waiting on the implementation of the
>> >>> >> > 'dist_info"
>> >>> >> > command
>> >>> >> > in the wheel project.
>> >>> >> > 2. Once that is done we can switch pip over to reading dist-info
>> >>> >> > rather
>> >>> >> > than
>> >>> >> > egg_info.
>> >>> >> > 3. Then we can move the backend over to setuptools. Because Jacob
>> >>> >> > has a
>> >>> >> > much
>> >>> >> > more efficient release system than pip, I anticipate having a
>> >>> >> > release of
>> >>> >> > setuptools first and then we can switch pip over to requiring a
>> >>> >> > newer
>> >>> >> > setuptools via PEP 518.
>> >>> >>
>> >>> >> I don't think pip actually has any use for the PEP 517
>> >>> >> prepare_wheel_metadata hook right now though? Historically
>> >>> >> 'setup.py
>> >>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
>> >>> >> install', but with a PEP 517 backend that's irrelevant because
>> >>> >> 'setup.py install' is never used. And in the future when pip has a
>> >>> >> real resolver, then prepare_wheel_metadata should allow some
>> >>> >> optimizations. But right now, prepare_wheel_metadata is completely
>> >>> >> useless AFAIK.
>> >>> >>
>> >>> >> So why is 'setup.py dist_info' a blocker for things?
>> >>> >>
>> >>> >> -n
>> >>> >>
>> >>> >> --
>> >>> >> Nathaniel J. Smith -- https://vorpus.org
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> 

Re: [Distutils] PEP 517 again

[xoviat]

The PR that I am taking about is not for pip but for the wheel project.

On Sep 4, 2017 8:19 PM, "Chris Jerdonek"  wrote:

> On Mon, Sep 4, 2017 at 6:08 PM, xoviat  wrote:
> > In any case, we're going to need this for prepare_metadata, so the
> question
> > you should ask is: what are the reasons for *not* merging this? I haven't
> > heard any so far but that doesn't mean that they don't exist. If there
> are
> > none, then I don't see why we cannot merge my wheel PR and do a release.
>
> FYI, you're not the only one waiting for PR's to be merged. There are
> 56 other PR's, some of which have already been approved by repo
> members with commit access. I would try to be a little more patient.
> Otherwise, everyone can be emailing the list saying the same thing and
> asking why their PR isn't being merged.
>
> --Chris
>
> >
> > 2017-09-04 19:51 GMT-05:00 xoviat :
> >>
> >> > The only reason I can think of that setuptools would need a dist_info
> >> command would be to implement the PEP 517 prepare_wheel_metadata hook.
> >>
> >> Yes. That is absolutely correct.
> >>
> >> > But this hook is optional and in fact provides no value right now, so
> >> it can't be a blocker for anything.
> >>
> >> The simplest way to start on this issue is to replace egg_info in pip
> with
> >> prepare_metadata_for_build_wheel. It is absolutely a historical
> artifact but
> >> I need to work on one issue at a time. The next issue will be replacing
> >> egg_info in pip with prepare_metadata_for_build_wheel.
> >>
> >> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith :
> >>>
> >>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat  wrote:
> >>> > Nathaniel:
> >>> >
> >>> > Pip requires egg_info to discover dependencies of source
> distributions
> >>> > so
> >>> > that it can build wheels all at once after downloading the
> >>> > requirements. I
> >>> > need to move pip off of egg_info as soon as possible and dist_info is
> >>> > required to do that.
> >>>
> >>> "Requires" is a strong word -- AFAIK this is just a historical
> >>> artifact. I don't really know what you're talking about in the second
> >>> sentence.
> >>>
> >>> The only reason I can think of that setuptools would need a dist_info
> >>> command would be to implement the PEP 517 prepare_wheel_metadata hook.
> >>> But this hook is optional and in fact provides no value right now, so
> >>> it can't be a blocker for anything. (In fact I can't see any reason
> >>> why pip would ever call it before the resolver lands.) So either (a)
> >>> there's some other reason you want a dist_info command, (b) there's
> >>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
> >>> of us is misunderstanding something :-).
> >>>
> >>> -n
> >>>
> >>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
> >>> >>
> >>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
> >>> >> > Just an update for everyone here:
> >>> >> >
> >>> >> > 1. We're currently waiting on the implementation of the
> 'dist_info"
> >>> >> > command
> >>> >> > in the wheel project.
> >>> >> > 2. Once that is done we can switch pip over to reading dist-info
> >>> >> > rather
> >>> >> > than
> >>> >> > egg_info.
> >>> >> > 3. Then we can move the backend over to setuptools. Because Jacob
> >>> >> > has a
> >>> >> > much
> >>> >> > more efficient release system than pip, I anticipate having a
> >>> >> > release of
> >>> >> > setuptools first and then we can switch pip over to requiring a
> >>> >> > newer
> >>> >> > setuptools via PEP 518.
> >>> >>
> >>> >> I don't think pip actually has any use for the PEP 517
> >>> >> prepare_wheel_metadata hook right now though? Historically 'setup.py
> >>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
> >>> >> install', but with a PEP 517 backend that's irrelevant because
> >>> >> 'setup.py install' is never used. And in the future when pip has a
> >>> >> real resolver, then prepare_wheel_metadata should allow some
> >>> >> optimizations. But right now, prepare_wheel_metadata is completely
> >>> >> useless AFAIK.
> >>> >>
> >>> >> So why is 'setup.py dist_info' a blocker for things?
> >>> >>
> >>> >> -n
> >>> >>
> >>> >> --
> >>> >> Nathaniel J. Smith -- https://vorpus.org
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Nathaniel J. Smith -- https://vorpus.org
> >>
> >>
> >
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Jerdonek]

On Mon, Sep 4, 2017 at 6:08 PM, xoviat  wrote:
> In any case, we're going to need this for prepare_metadata, so the question
> you should ask is: what are the reasons for *not* merging this? I haven't
> heard any so far but that doesn't mean that they don't exist. If there are
> none, then I don't see why we cannot merge my wheel PR and do a release.

FYI, you're not the only one waiting for PR's to be merged. There are
56 other PR's, some of which have already been approved by repo
members with commit access. I would try to be a little more patient.
Otherwise, everyone can be emailing the list saying the same thing and
asking why their PR isn't being merged.

--Chris

>
> 2017-09-04 19:51 GMT-05:00 xoviat :
>>
>> > The only reason I can think of that setuptools would need a dist_info
>> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>>
>> Yes. That is absolutely correct.
>>
>> > But this hook is optional and in fact provides no value right now, so
>> it can't be a blocker for anything.
>>
>> The simplest way to start on this issue is to replace egg_info in pip with
>> prepare_metadata_for_build_wheel. It is absolutely a historical artifact but
>> I need to work on one issue at a time. The next issue will be replacing
>> egg_info in pip with prepare_metadata_for_build_wheel.
>>
>> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith :
>>>
>>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat  wrote:
>>> > Nathaniel:
>>> >
>>> > Pip requires egg_info to discover dependencies of source distributions
>>> > so
>>> > that it can build wheels all at once after downloading the
>>> > requirements. I
>>> > need to move pip off of egg_info as soon as possible and dist_info is
>>> > required to do that.
>>>
>>> "Requires" is a strong word -- AFAIK this is just a historical
>>> artifact. I don't really know what you're talking about in the second
>>> sentence.
>>>
>>> The only reason I can think of that setuptools would need a dist_info
>>> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>>> But this hook is optional and in fact provides no value right now, so
>>> it can't be a blocker for anything. (In fact I can't see any reason
>>> why pip would ever call it before the resolver lands.) So either (a)
>>> there's some other reason you want a dist_info command, (b) there's
>>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
>>> of us is misunderstanding something :-).
>>>
>>> -n
>>>
>>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
>>> >>
>>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
>>> >> > Just an update for everyone here:
>>> >> >
>>> >> > 1. We're currently waiting on the implementation of the 'dist_info"
>>> >> > command
>>> >> > in the wheel project.
>>> >> > 2. Once that is done we can switch pip over to reading dist-info
>>> >> > rather
>>> >> > than
>>> >> > egg_info.
>>> >> > 3. Then we can move the backend over to setuptools. Because Jacob
>>> >> > has a
>>> >> > much
>>> >> > more efficient release system than pip, I anticipate having a
>>> >> > release of
>>> >> > setuptools first and then we can switch pip over to requiring a
>>> >> > newer
>>> >> > setuptools via PEP 518.
>>> >>
>>> >> I don't think pip actually has any use for the PEP 517
>>> >> prepare_wheel_metadata hook right now though? Historically 'setup.py
>>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
>>> >> install', but with a PEP 517 backend that's irrelevant because
>>> >> 'setup.py install' is never used. And in the future when pip has a
>>> >> real resolver, then prepare_wheel_metadata should allow some
>>> >> optimizations. But right now, prepare_wheel_metadata is completely
>>> >> useless AFAIK.
>>> >>
>>> >> So why is 'setup.py dist_info' a blocker for things?
>>> >>
>>> >> -n
>>> >>
>>> >> --
>>> >> Nathaniel J. Smith -- https://vorpus.org
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Nathaniel J. Smith -- https://vorpus.org
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Mon, Sep 4, 2017 at 5:51 PM, xoviat  wrote:
>> The only reason I can think of that setuptools would need a dist_info
> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>
> Yes. That is absolutely correct.
>
>> But this hook is optional and in fact provides no value right now, so
> it can't be a blocker for anything.
>
> The simplest way to start on this issue is to replace egg_info in pip with
> prepare_metadata_for_build_wheel. It is absolutely a historical artifact but
> I need to work on one issue at a time. The next issue will be replacing
> egg_info in pip with prepare_metadata_for_build_wheel.

This still doesn't make sense to me.

To support PEP 517, pip HAS to support backends that don't provide
prepare_metadata_for_build_wheel. You will have to handle this before
PEP 517 support can ship. So what's your plan for after you replace
egg_info with prepare_metadata_for_build_wheel? Turning around and
deleting the code you just wrote?

-n

> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith :
>>
>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat  wrote:
>> > Nathaniel:
>> >
>> > Pip requires egg_info to discover dependencies of source distributions
>> > so
>> > that it can build wheels all at once after downloading the requirements.
>> > I
>> > need to move pip off of egg_info as soon as possible and dist_info is
>> > required to do that.
>>
>> "Requires" is a strong word -- AFAIK this is just a historical
>> artifact. I don't really know what you're talking about in the second
>> sentence.
>>
>> The only reason I can think of that setuptools would need a dist_info
>> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>> But this hook is optional and in fact provides no value right now, so
>> it can't be a blocker for anything. (In fact I can't see any reason
>> why pip would ever call it before the resolver lands.) So either (a)
>> there's some other reason you want a dist_info command, (b) there's
>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
>> of us is misunderstanding something :-).
>>
>> -n
>>
>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
>> >>
>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
>> >> > Just an update for everyone here:
>> >> >
>> >> > 1. We're currently waiting on the implementation of the 'dist_info"
>> >> > command
>> >> > in the wheel project.
>> >> > 2. Once that is done we can switch pip over to reading dist-info
>> >> > rather
>> >> > than
>> >> > egg_info.
>> >> > 3. Then we can move the backend over to setuptools. Because Jacob has
>> >> > a
>> >> > much
>> >> > more efficient release system than pip, I anticipate having a release
>> >> > of
>> >> > setuptools first and then we can switch pip over to requiring a newer
>> >> > setuptools via PEP 518.
>> >>
>> >> I don't think pip actually has any use for the PEP 517
>> >> prepare_wheel_metadata hook right now though? Historically 'setup.py
>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
>> >> install', but with a PEP 517 backend that's irrelevant because
>> >> 'setup.py install' is never used. And in the future when pip has a
>> >> real resolver, then prepare_wheel_metadata should allow some
>> >> optimizations. But right now, prepare_wheel_metadata is completely
>> >> useless AFAIK.
>> >>
>> >> So why is 'setup.py dist_info' a blocker for things?
>> >>
>> >> -n
>> >>
>> >> --
>> >> Nathaniel J. Smith -- https://vorpus.org
>> >
>> >
>>
>>
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>
>



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

In any case, we're going to need this for prepare_metadata, so the question
you should ask is: what are the reasons for *not* merging this? I haven't
heard any so far but that doesn't mean that they don't exist. If there are
none, then I don't see why we cannot merge my wheel PR and do a release.

2017-09-04 19:51 GMT-05:00 xoviat :

> > The only reason I can think of that setuptools would need a dist_info
> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>
> Yes. That is absolutely correct.
>
> > But this hook is optional and in fact provides no value right now, so
> it can't be a blocker for anything.
>
> The simplest way to start on this issue is to replace egg_info in pip with
> prepare_metadata_for_build_wheel. It is absolutely a historical artifact
> but I need to work on one issue at a time. The next issue will be replacing
> egg_info in pip with prepare_metadata_for_build_wheel.
>
> 2017-09-04 19:34 GMT-05:00 Nathaniel Smith :
>
>> On Mon, Sep 4, 2017 at 5:09 PM, xoviat  wrote:
>> > Nathaniel:
>> >
>> > Pip requires egg_info to discover dependencies of source distributions
>> so
>> > that it can build wheels all at once after downloading the
>> requirements. I
>> > need to move pip off of egg_info as soon as possible and dist_info is
>> > required to do that.
>>
>> "Requires" is a strong word -- AFAIK this is just a historical
>> artifact. I don't really know what you're talking about in the second
>> sentence.
>>
>> The only reason I can think of that setuptools would need a dist_info
>> command would be to implement the PEP 517 prepare_wheel_metadata hook.
>> But this hook is optional and in fact provides no value right now, so
>> it can't be a blocker for anything. (In fact I can't see any reason
>> why pip would ever call it before the resolver lands.) So either (a)
>> there's some other reason you want a dist_info command, (b) there's
>> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
>> of us is misunderstanding something :-).
>>
>> -n
>>
>> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
>> >>
>> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
>> >> > Just an update for everyone here:
>> >> >
>> >> > 1. We're currently waiting on the implementation of the 'dist_info"
>> >> > command
>> >> > in the wheel project.
>> >> > 2. Once that is done we can switch pip over to reading dist-info
>> rather
>> >> > than
>> >> > egg_info.
>> >> > 3. Then we can move the backend over to setuptools. Because Jacob
>> has a
>> >> > much
>> >> > more efficient release system than pip, I anticipate having a
>> release of
>> >> > setuptools first and then we can switch pip over to requiring a newer
>> >> > setuptools via PEP 518.
>> >>
>> >> I don't think pip actually has any use for the PEP 517
>> >> prepare_wheel_metadata hook right now though? Historically 'setup.py
>> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
>> >> install', but with a PEP 517 backend that's irrelevant because
>> >> 'setup.py install' is never used. And in the future when pip has a
>> >> real resolver, then prepare_wheel_metadata should allow some
>> >> optimizations. But right now, prepare_wheel_metadata is completely
>> >> useless AFAIK.
>> >>
>> >> So why is 'setup.py dist_info' a blocker for things?
>> >>
>> >> -n
>> >>
>> >> --
>> >> Nathaniel J. Smith -- https://vorpus.org
>> >
>> >
>>
>>
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> The only reason I can think of that setuptools would need a dist_info
command would be to implement the PEP 517 prepare_wheel_metadata hook.

Yes. That is absolutely correct.

> But this hook is optional and in fact provides no value right now, so
it can't be a blocker for anything.

The simplest way to start on this issue is to replace egg_info in pip with
prepare_metadata_for_build_wheel. It is absolutely a historical artifact
but I need to work on one issue at a time. The next issue will be replacing
egg_info in pip with prepare_metadata_for_build_wheel.

2017-09-04 19:34 GMT-05:00 Nathaniel Smith :

> On Mon, Sep 4, 2017 at 5:09 PM, xoviat  wrote:
> > Nathaniel:
> >
> > Pip requires egg_info to discover dependencies of source distributions so
> > that it can build wheels all at once after downloading the requirements.
> I
> > need to move pip off of egg_info as soon as possible and dist_info is
> > required to do that.
>
> "Requires" is a strong word -- AFAIK this is just a historical
> artifact. I don't really know what you're talking about in the second
> sentence.
>
> The only reason I can think of that setuptools would need a dist_info
> command would be to implement the PEP 517 prepare_wheel_metadata hook.
> But this hook is optional and in fact provides no value right now, so
> it can't be a blocker for anything. (In fact I can't see any reason
> why pip would ever call it before the resolver lands.) So either (a)
> there's some other reason you want a dist_info command, (b) there's
> some reason I'm missing why prepare_wheel_metadata matters, or (c) one
> of us is misunderstanding something :-).
>
> -n
>
> > 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
> >>
> >> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
> >> > Just an update for everyone here:
> >> >
> >> > 1. We're currently waiting on the implementation of the 'dist_info"
> >> > command
> >> > in the wheel project.
> >> > 2. Once that is done we can switch pip over to reading dist-info
> rather
> >> > than
> >> > egg_info.
> >> > 3. Then we can move the backend over to setuptools. Because Jacob has
> a
> >> > much
> >> > more efficient release system than pip, I anticipate having a release
> of
> >> > setuptools first and then we can switch pip over to requiring a newer
> >> > setuptools via PEP 518.
> >>
> >> I don't think pip actually has any use for the PEP 517
> >> prepare_wheel_metadata hook right now though? Historically 'setup.py
> >> egg-info' was needed to kluge around unwanted behavior in 'setup.py
> >> install', but with a PEP 517 backend that's irrelevant because
> >> 'setup.py install' is never used. And in the future when pip has a
> >> real resolver, then prepare_wheel_metadata should allow some
> >> optimizations. But right now, prepare_wheel_metadata is completely
> >> useless AFAIK.
> >>
> >> So why is 'setup.py dist_info' a blocker for things?
> >>
> >> -n
> >>
> >> --
> >> Nathaniel J. Smith -- https://vorpus.org
> >
> >
>
>
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Mon, Sep 4, 2017 at 5:09 PM, xoviat  wrote:
> Nathaniel:
>
> Pip requires egg_info to discover dependencies of source distributions so
> that it can build wheels all at once after downloading the requirements. I
> need to move pip off of egg_info as soon as possible and dist_info is
> required to do that.

"Requires" is a strong word -- AFAIK this is just a historical
artifact. I don't really know what you're talking about in the second
sentence.

The only reason I can think of that setuptools would need a dist_info
command would be to implement the PEP 517 prepare_wheel_metadata hook.
But this hook is optional and in fact provides no value right now, so
it can't be a blocker for anything. (In fact I can't see any reason
why pip would ever call it before the resolver lands.) So either (a)
there's some other reason you want a dist_info command, (b) there's
some reason I'm missing why prepare_wheel_metadata matters, or (c) one
of us is misunderstanding something :-).

-n

> 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
>>
>> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
>> > Just an update for everyone here:
>> >
>> > 1. We're currently waiting on the implementation of the 'dist_info"
>> > command
>> > in the wheel project.
>> > 2. Once that is done we can switch pip over to reading dist-info rather
>> > than
>> > egg_info.
>> > 3. Then we can move the backend over to setuptools. Because Jacob has a
>> > much
>> > more efficient release system than pip, I anticipate having a release of
>> > setuptools first and then we can switch pip over to requiring a newer
>> > setuptools via PEP 518.
>>
>> I don't think pip actually has any use for the PEP 517
>> prepare_wheel_metadata hook right now though? Historically 'setup.py
>> egg-info' was needed to kluge around unwanted behavior in 'setup.py
>> install', but with a PEP 517 backend that's irrelevant because
>> 'setup.py install' is never used. And in the future when pip has a
>> real resolver, then prepare_wheel_metadata should allow some
>> optimizations. But right now, prepare_wheel_metadata is completely
>> useless AFAIK.
>>
>> So why is 'setup.py dist_info' a blocker for things?
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>
>



-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Also if someone with pip write access could please discuss and hopefully
merge my initial PR on pip, I would very much appreciate it. Paul seems to
be short on time.

2017-09-04 19:09 GMT-05:00 xoviat :

> Nathaniel:
>
> Pip requires egg_info to discover dependencies of source distributions so
> that it can build wheels all at once after downloading the requirements. I
> need to move pip off of egg_info as soon as possible and dist_info is
> required to do that.
>
> 2017-09-03 21:00 GMT-05:00 Nathaniel Smith :
>
>> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
>> > Just an update for everyone here:
>> >
>> > 1. We're currently waiting on the implementation of the 'dist_info"
>> command
>> > in the wheel project.
>> > 2. Once that is done we can switch pip over to reading dist-info rather
>> than
>> > egg_info.
>> > 3. Then we can move the backend over to setuptools. Because Jacob has a
>> much
>> > more efficient release system than pip, I anticipate having a release of
>> > setuptools first and then we can switch pip over to requiring a newer
>> > setuptools via PEP 518.
>>
>> I don't think pip actually has any use for the PEP 517
>> prepare_wheel_metadata hook right now though? Historically 'setup.py
>> egg-info' was needed to kluge around unwanted behavior in 'setup.py
>> install', but with a PEP 517 backend that's irrelevant because
>> 'setup.py install' is never used. And in the future when pip has a
>> real resolver, then prepare_wheel_metadata should allow some
>> optimizations. But right now, prepare_wheel_metadata is completely
>> useless AFAIK.
>>
>> So why is 'setup.py dist_info' a blocker for things?
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Nathaniel:

Pip requires egg_info to discover dependencies of source distributions so
that it can build wheels all at once after downloading the requirements. I
need to move pip off of egg_info as soon as possible and dist_info is
required to do that.

2017-09-03 21:00 GMT-05:00 Nathaniel Smith :

> On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
> > Just an update for everyone here:
> >
> > 1. We're currently waiting on the implementation of the 'dist_info"
> command
> > in the wheel project.
> > 2. Once that is done we can switch pip over to reading dist-info rather
> than
> > egg_info.
> > 3. Then we can move the backend over to setuptools. Because Jacob has a
> much
> > more efficient release system than pip, I anticipate having a release of
> > setuptools first and then we can switch pip over to requiring a newer
> > setuptools via PEP 518.
>
> I don't think pip actually has any use for the PEP 517
> prepare_wheel_metadata hook right now though? Historically 'setup.py
> egg-info' was needed to kluge around unwanted behavior in 'setup.py
> install', but with a PEP 517 backend that's irrelevant because
> 'setup.py install' is never used. And in the future when pip has a
> real resolver, then prepare_wheel_metadata should allow some
> optimizations. But right now, prepare_wheel_metadata is completely
> useless AFAIK.
>
> So why is 'setup.py dist_info' a blocker for things?
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nick Coghlan]

On 3 September 2017 at 05:42, Donald Stufft  wrote:
> On Sep 1, 2017, at 2:30 PM, Chris Barker  wrote:
>> Do the Linux distros use pip to build their packages?
>
> Not that I am aware of.

Fedora's build macros for Python projects currently rely on running
setup.py directly, but we've been considering switching to pip instead
since 2013 or so. PEP 517 is likely to provide the impetus to switch
from "maybe we should do that" to "we need to do that, at least if
setup.py is missing, and potentially always, so we get more consistent
installation metadata"

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Sun, Sep 3, 2017 at 11:14 AM, xoviat  wrote:
> Just an update for everyone here:
>
> 1. We're currently waiting on the implementation of the 'dist_info" command
> in the wheel project.
> 2. Once that is done we can switch pip over to reading dist-info rather than
> egg_info.
> 3. Then we can move the backend over to setuptools. Because Jacob has a much
> more efficient release system than pip, I anticipate having a release of
> setuptools first and then we can switch pip over to requiring a newer
> setuptools via PEP 518.

I don't think pip actually has any use for the PEP 517
prepare_wheel_metadata hook right now though? Historically 'setup.py
egg-info' was needed to kluge around unwanted behavior in 'setup.py
install', but with a PEP 517 backend that's irrelevant because
'setup.py install' is never used. And in the future when pip has a
real resolver, then prepare_wheel_metadata should allow some
optimizations. But right now, prepare_wheel_metadata is completely
useless AFAIK.

So why is 'setup.py dist_info' a blocker for things?

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[cyanrave]

Great discussions on this PEP over the proposal, here and in the archives. I've 
just now gotten plugged in to the distutils distro, at a seemingly perfect 
time... somewhat new to Python packaging and still learning.

Many of the issues observed by users in the mail chain are reasons why we had 
marching orders at the organization I work for to internally build all whls, on 
the platforms we support by using 'pip wheel' with specific flags. One of these 
flags is to pull source only (pip download, no-binary) and direct 'pip wheel' 
to build with that source dir.

For the caching issue we ended up using Docker, spinning up a new instance per 
build with the platforms we support, running the build, and uploading those 
artifacts to our internal repo for pulls by the users, then thrashing that 
container wholly. In a similar way, I see the 'venv' requirement of 517 
mirroring this process.

The presupposition of this pipeline we built is that metadata and source repo 
structures are trusted to produce accurate binaries... the more I learn about 
how projects / packages have set up their build instructions, the more 
horrified I become in the variance of the implementations out in the wild.

517 and 518 to me seems like a way to reign in some of these variances and 
formalize them, especially around setup.py and metadata... Is that correct or 
am I way off base with the discussion and proposal so far?

Best regards,
C. Schmautz

> On Sep 2, 2017, at 12:42 PM, Donald Stufft  wrote:
> 
> 
>> On Sep 1, 2017, at 2:30 PM, Chris Barker  wrote:
>> 
>>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan  wrote:
>>> 
>>> since it
>>> doesn't reliably distinguish between "this cached wheel was downloaded
>>> from a repository" and "this wheel was generated locally with a
>>> particular version of Python".
>> 
>> It shouldn't have to. sigh.
>> 
>>>  PEP 517 deliberately doesn't let
>>> frontends do that as part of the initial build process (instead, if
>>> they want to adjust the tags, they need to do it as a post-processing
>>> step).
>>> 
>>> Since PEP 517 breaks the current workaround for the caching scheme
>>> being inaccurate, the most suitable response is to instead fix pip's
>>> caching scheme to use a two tier local cache:
>> 
>> I'm still confused -- if setuptools ( invoked  by pip) is producing 
>> incorrectly named wheels -- surely that's a bug-fix/workaround that should 
>> go into setuptools?
>> 
>> If the build is being run by pip, then doesn't setuptools have all the info 
>> about the system that pip has?
>> 
>> 
> 
> 
> Someone building a wheel for distribution is likely intimately aware of that 
> project, and can take care to ensure that the wheel is built in such a way 
> that it is giving people the most optimal behavior. Pip is auto building 
> wheels without human intervention, and as such there is nobody there to make 
> sure that we’re not accidentally creating a too-broad wheel, so we want to 
> ensure that we have some mechanism in place for not re-using the wheel across 
> boundaries that might cause issues.
> 
> 
>> 
>>> we also have plenty of PyPI users that
>>> explicitly *opt out* of using publisher-provided pre-built binaries.
>>> While Linux distributions are the most common example (see [1] for
>>> Fedora's policy, for example), we're not the only ones that have that
>>> kind of rule in place.
>> 
>> But this is an argument for why pypi should host sdists, and the build tools 
>> should build sdists, but not why pip should auto-build them.
>> 
>> Condo-forge, for example, almost always builds from source -- sometimes an 
>> sdist 
>> from pypi, sometimes a source distribution from github or wherever the 
>> package is hosted. And sometimes from a git tag ( last resort).
> 
> 
> Pip supports more systems than Conda does, and we do that by relying on auto 
> building support. Pip supports systems that don’t have a wheel compatibility 
> tag defined for them, and for which we’re unlikely to ever have any wheels 
> published for (much less wide spread). It’s pretty easy to cover 
> Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, 
> OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.
> 
> Pip works in all these situations, and it does so by relying on building from 
> source.
> 
> 
>> 
>> Do the Linux distros use pip to build their packages?
> 
> 
> Not that I am aware of.
> 
>> 
>> I tried to do that with conda-packages, and failed due to pip's caching  
>> behavior-- it probably would have worked fine in production, but when I was 
>> developing the build script, I couldn't reliably get pip to ignore cached 
>> wheels from previous experimental builds.
> 
> 
> Adding —no-cache-dir disables all of pip’s caching.
> 
> —
> Donald Stufft
> 
> 
> 
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> 

Re: [Distutils] PEP 517 again

[xoviat]

Just an update for everyone here:

1. We're currently waiting on the implementation of the 'dist_info" command
in the wheel project.
2. Once that is done we can switch pip over to reading dist-info rather
than egg_info.
3. Then we can move the backend over to setuptools. Because Jacob has a
much more efficient release system than pip, I anticipate having a release
of setuptools first and then we can switch pip over to requiring a newer
setuptools via PEP 518.

2017-09-02 19:51 GMT-05:00 Chris Jerdonek :

> On Sat, Sep 2, 2017 at 5:17 PM xoviat  wrote:
>
>> Whatever it was, removing it seems to have had no effect on the tests. I
>> will remove it unless someone has an objection.
>>
>
> Just FYI, I wouldn't take the tests still passing as a major signal. I've
> noticed there are even common code paths / functional scenarios that aren't
> under test.
>
> --Chris
>
>
>
>
>> 2017-09-02 18:26 GMT-05:00 xoviat :
>>
>>> Donald,
>>>
>>>
>>> This was your work in https://github.com/pypa/pip/pull/2169.
>>> Unfortunately the comments were quite sparse.
>>>
>>> 2017-09-02 18:25 GMT-05:00 xoviat :
>>>
 One more issue that has come up is that "--no-user-cfg" seems to be
 passed to the egg_info invocation if the "isolated" parameter is enabled. I
 don't understand what this does, but it is again not defined in the PEP 517
 interface. Should we always pass this parameter or should we never pass it?

 2017-09-02 14:42 GMT-05:00 Donald Stufft :

>
> On Sep 1, 2017, at 2:30 PM, Chris Barker 
> wrote:
>
> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan 
> wrote:
>
> since it
>> doesn't reliably distinguish between "this cached wheel was downloaded
>> from a repository" and "this wheel was generated locally with a
>> particular version of Python".
>
>
> It shouldn't have to. sigh.
>
>>
>  PEP 517 deliberately doesn't let
>> frontends do that as part of the initial build process (instead, if
>> they want to adjust the tags, they need to do it as a post-processing
>> step).
>>
>> Since PEP 517 breaks the current workaround for the caching scheme
>> being inaccurate, the most suitable response is to instead fix pip's
>> caching scheme to use a two tier local cache:
>
>
> I'm still confused -- if setuptools ( invoked  by pip) is producing
> incorrectly named wheels -- surely that's a bug-fix/workaround that should
> go into setuptools?
>
> If the build is being run by pip, then doesn't setuptools have all the
> info about the system that pip has?
>
>
>
>
> Someone building a wheel for distribution is likely intimately aware
> of that project, and can take care to ensure that the wheel is built in
> such a way that it is giving people the most optimal behavior. Pip is auto
> building wheels without human intervention, and as such there is nobody
> there to make sure that we’re not accidentally creating a too-broad wheel,
> so we want to ensure that we have some mechanism in place for not re-using
> the wheel across boundaries that might cause issues.
>
>
>
> we also have plenty of PyPI users that
>> explicitly *opt out* of using publisher-provided pre-built binaries.
>> While Linux distributions are the most common example (see [1] for
>> Fedora's policy, for example), we're not the only ones that have that
>> kind of rule in place.
>
>
> But this is an argument for why pypi should host sdists, and the build
> tools should build sdists, but not why pip should auto-build them.
>
>>
> Condo-forge, for example, almost always builds from source --
> sometimes an sdist
> from pypi, sometimes a source distribution from github or wherever the
> package is hosted. And sometimes from a git tag ( last resort).
>
>
>
> Pip supports more systems than Conda does, and we do that by relying
> on auto building support. Pip supports systems that don’t have a wheel
> compatibility tag defined for them, and for which we’re unlikely to ever
> have any wheels published for (much less wide spread). It’s pretty easy to
> cover Windows/macOS/Some Linux systems, but when you start talking about
> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets
> extremely long.
>
> Pip works in all these situations, and it does so by relying on
> building from source.
>
>
>
> Do the Linux distros use pip to build their packages?
>
>
>
> Not that I am aware of.
>
>
> I tried to do that with conda-packages, and failed due to pip's
> caching  behavior-- it probably would have worked fine in production, but
> when I was developing the build script, I 

Re: [Distutils] PEP 517 again

[Chris Jerdonek]

On Sat, Sep 2, 2017 at 5:17 PM xoviat  wrote:

> Whatever it was, removing it seems to have had no effect on the tests. I
> will remove it unless someone has an objection.
>

Just FYI, I wouldn't take the tests still passing as a major signal. I've
noticed there are even common code paths / functional scenarios that aren't
under test.

--Chris




> 2017-09-02 18:26 GMT-05:00 xoviat :
>
>> Donald,
>>
>>
>> This was your work in https://github.com/pypa/pip/pull/2169.
>> Unfortunately the comments were quite sparse.
>>
>> 2017-09-02 18:25 GMT-05:00 xoviat :
>>
>>> One more issue that has come up is that "--no-user-cfg" seems to be
>>> passed to the egg_info invocation if the "isolated" parameter is enabled. I
>>> don't understand what this does, but it is again not defined in the PEP 517
>>> interface. Should we always pass this parameter or should we never pass it?
>>>
>>> 2017-09-02 14:42 GMT-05:00 Donald Stufft :
>>>

 On Sep 1, 2017, at 2:30 PM, Chris Barker  wrote:

 On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan 
 wrote:

 since it
> doesn't reliably distinguish between "this cached wheel was downloaded
> from a repository" and "this wheel was generated locally with a
> particular version of Python".


 It shouldn't have to. sigh.

>
  PEP 517 deliberately doesn't let
> frontends do that as part of the initial build process (instead, if
> they want to adjust the tags, they need to do it as a post-processing
> step).
>
> Since PEP 517 breaks the current workaround for the caching scheme
> being inaccurate, the most suitable response is to instead fix pip's
> caching scheme to use a two tier local cache:


 I'm still confused -- if setuptools ( invoked  by pip) is producing
 incorrectly named wheels -- surely that's a bug-fix/workaround that should
 go into setuptools?

 If the build is being run by pip, then doesn't setuptools have all the
 info about the system that pip has?




 Someone building a wheel for distribution is likely intimately aware of
 that project, and can take care to ensure that the wheel is built in such a
 way that it is giving people the most optimal behavior. Pip is auto
 building wheels without human intervention, and as such there is nobody
 there to make sure that we’re not accidentally creating a too-broad wheel,
 so we want to ensure that we have some mechanism in place for not re-using
 the wheel across boundaries that might cause issues.



 we also have plenty of PyPI users that
> explicitly *opt out* of using publisher-provided pre-built binaries.
> While Linux distributions are the most common example (see [1] for
> Fedora's policy, for example), we're not the only ones that have that
> kind of rule in place.


 But this is an argument for why pypi should host sdists, and the build
 tools should build sdists, but not why pip should auto-build them.

>
 Condo-forge, for example, almost always builds from source -- sometimes
 an sdist
 from pypi, sometimes a source distribution from github or wherever the
 package is hosted. And sometimes from a git tag ( last resort).



 Pip supports more systems than Conda does, and we do that by relying on
 auto building support. Pip supports systems that don’t have a wheel
 compatibility tag defined for them, and for which we’re unlikely to ever
 have any wheels published for (much less wide spread). It’s pretty easy to
 cover Windows/macOS/Some Linux systems, but when you start talking about
 FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets
 extremely long.

 Pip works in all these situations, and it does so by relying on
 building from source.



 Do the Linux distros use pip to build their packages?



 Not that I am aware of.


 I tried to do that with conda-packages, and failed due to pip's caching
  behavior-- it probably would have worked fine in production, but when I
 was developing the build script, I couldn't reliably get pip to ignore
 cached wheels from previous experimental builds.



 Adding —no-cache-dir disables all of pip’s caching.

 —
 Donald Stufft




 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig


>>>
>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org

Re: [Distutils] PEP 517 again

[xoviat]

Whatever it was, removing it seems to have had no effect on the tests. I
will remove it unless someone has an objection.

2017-09-02 18:26 GMT-05:00 xoviat :

> Donald,
>
>
> This was your work in https://github.com/pypa/pip/pull/2169.
> Unfortunately the comments were quite sparse.
>
> 2017-09-02 18:25 GMT-05:00 xoviat :
>
>> One more issue that has come up is that "--no-user-cfg" seems to be
>> passed to the egg_info invocation if the "isolated" parameter is enabled. I
>> don't understand what this does, but it is again not defined in the PEP 517
>> interface. Should we always pass this parameter or should we never pass it?
>>
>> 2017-09-02 14:42 GMT-05:00 Donald Stufft :
>>
>>>
>>> On Sep 1, 2017, at 2:30 PM, Chris Barker  wrote:
>>>
>>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan  wrote:
>>>
>>> since it
 doesn't reliably distinguish between "this cached wheel was downloaded
 from a repository" and "this wheel was generated locally with a
 particular version of Python".
>>>
>>>
>>> It shouldn't have to. sigh.
>>>

>>>  PEP 517 deliberately doesn't let
 frontends do that as part of the initial build process (instead, if
 they want to adjust the tags, they need to do it as a post-processing
 step).

 Since PEP 517 breaks the current workaround for the caching scheme
 being inaccurate, the most suitable response is to instead fix pip's
 caching scheme to use a two tier local cache:
>>>
>>>
>>> I'm still confused -- if setuptools ( invoked  by pip) is producing
>>> incorrectly named wheels -- surely that's a bug-fix/workaround that should
>>> go into setuptools?
>>>
>>> If the build is being run by pip, then doesn't setuptools have all the
>>> info about the system that pip has?
>>>
>>>
>>>
>>>
>>> Someone building a wheel for distribution is likely intimately aware of
>>> that project, and can take care to ensure that the wheel is built in such a
>>> way that it is giving people the most optimal behavior. Pip is auto
>>> building wheels without human intervention, and as such there is nobody
>>> there to make sure that we’re not accidentally creating a too-broad wheel,
>>> so we want to ensure that we have some mechanism in place for not re-using
>>> the wheel across boundaries that might cause issues.
>>>
>>>
>>>
>>> we also have plenty of PyPI users that
 explicitly *opt out* of using publisher-provided pre-built binaries.
 While Linux distributions are the most common example (see [1] for
 Fedora's policy, for example), we're not the only ones that have that
 kind of rule in place.
>>>
>>>
>>> But this is an argument for why pypi should host sdists, and the build
>>> tools should build sdists, but not why pip should auto-build them.
>>>

>>> Condo-forge, for example, almost always builds from source -- sometimes
>>> an sdist
>>> from pypi, sometimes a source distribution from github or wherever the
>>> package is hosted. And sometimes from a git tag ( last resort).
>>>
>>>
>>>
>>> Pip supports more systems than Conda does, and we do that by relying on
>>> auto building support. Pip supports systems that don’t have a wheel
>>> compatibility tag defined for them, and for which we’re unlikely to ever
>>> have any wheels published for (much less wide spread). It’s pretty easy to
>>> cover Windows/macOS/Some Linux systems, but when you start talking about
>>> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets
>>> extremely long.
>>>
>>> Pip works in all these situations, and it does so by relying on building
>>> from source.
>>>
>>>
>>>
>>> Do the Linux distros use pip to build their packages?
>>>
>>>
>>>
>>> Not that I am aware of.
>>>
>>>
>>> I tried to do that with conda-packages, and failed due to pip's caching
>>>  behavior-- it probably would have worked fine in production, but when I
>>> was developing the build script, I couldn't reliably get pip to ignore
>>> cached wheels from previous experimental builds.
>>>
>>>
>>>
>>> Adding —no-cache-dir disables all of pip’s caching.
>>>
>>> —
>>> Donald Stufft
>>>
>>>
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Donald,


This was your work in https://github.com/pypa/pip/pull/2169. Unfortunately
the comments were quite sparse.

2017-09-02 18:25 GMT-05:00 xoviat :

> One more issue that has come up is that "--no-user-cfg" seems to be passed
> to the egg_info invocation if the "isolated" parameter is enabled. I don't
> understand what this does, but it is again not defined in the PEP 517
> interface. Should we always pass this parameter or should we never pass it?
>
> 2017-09-02 14:42 GMT-05:00 Donald Stufft :
>
>>
>> On Sep 1, 2017, at 2:30 PM, Chris Barker  wrote:
>>
>> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan  wrote:
>>
>> since it
>>> doesn't reliably distinguish between "this cached wheel was downloaded
>>> from a repository" and "this wheel was generated locally with a
>>> particular version of Python".
>>
>>
>> It shouldn't have to. sigh.
>>
>>>
>>  PEP 517 deliberately doesn't let
>>> frontends do that as part of the initial build process (instead, if
>>> they want to adjust the tags, they need to do it as a post-processing
>>> step).
>>>
>>> Since PEP 517 breaks the current workaround for the caching scheme
>>> being inaccurate, the most suitable response is to instead fix pip's
>>> caching scheme to use a two tier local cache:
>>
>>
>> I'm still confused -- if setuptools ( invoked  by pip) is producing
>> incorrectly named wheels -- surely that's a bug-fix/workaround that should
>> go into setuptools?
>>
>> If the build is being run by pip, then doesn't setuptools have all the
>> info about the system that pip has?
>>
>>
>>
>>
>> Someone building a wheel for distribution is likely intimately aware of
>> that project, and can take care to ensure that the wheel is built in such a
>> way that it is giving people the most optimal behavior. Pip is auto
>> building wheels without human intervention, and as such there is nobody
>> there to make sure that we’re not accidentally creating a too-broad wheel,
>> so we want to ensure that we have some mechanism in place for not re-using
>> the wheel across boundaries that might cause issues.
>>
>>
>>
>> we also have plenty of PyPI users that
>>> explicitly *opt out* of using publisher-provided pre-built binaries.
>>> While Linux distributions are the most common example (see [1] for
>>> Fedora's policy, for example), we're not the only ones that have that
>>> kind of rule in place.
>>
>>
>> But this is an argument for why pypi should host sdists, and the build
>> tools should build sdists, but not why pip should auto-build them.
>>
>>>
>> Condo-forge, for example, almost always builds from source -- sometimes
>> an sdist
>> from pypi, sometimes a source distribution from github or wherever the
>> package is hosted. And sometimes from a git tag ( last resort).
>>
>>
>>
>> Pip supports more systems than Conda does, and we do that by relying on
>> auto building support. Pip supports systems that don’t have a wheel
>> compatibility tag defined for them, and for which we’re unlikely to ever
>> have any wheels published for (much less wide spread). It’s pretty easy to
>> cover Windows/macOS/Some Linux systems, but when you start talking about
>> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets
>> extremely long.
>>
>> Pip works in all these situations, and it does so by relying on building
>> from source.
>>
>>
>>
>> Do the Linux distros use pip to build their packages?
>>
>>
>>
>> Not that I am aware of.
>>
>>
>> I tried to do that with conda-packages, and failed due to pip's caching
>>  behavior-- it probably would have worked fine in production, but when I
>> was developing the build script, I couldn't reliably get pip to ignore
>> cached wheels from previous experimental builds.
>>
>>
>>
>> Adding —no-cache-dir disables all of pip’s caching.
>>
>> —
>> Donald Stufft
>>
>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

One more issue that has come up is that "--no-user-cfg" seems to be passed
to the egg_info invocation if the "isolated" parameter is enabled. I don't
understand what this does, but it is again not defined in the PEP 517
interface. Should we always pass this parameter or should we never pass it?

2017-09-02 14:42 GMT-05:00 Donald Stufft :

>
> On Sep 1, 2017, at 2:30 PM, Chris Barker  wrote:
>
> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan  wrote:
>
> since it
>> doesn't reliably distinguish between "this cached wheel was downloaded
>> from a repository" and "this wheel was generated locally with a
>> particular version of Python".
>
>
> It shouldn't have to. sigh.
>
>>
>  PEP 517 deliberately doesn't let
>> frontends do that as part of the initial build process (instead, if
>> they want to adjust the tags, they need to do it as a post-processing
>> step).
>>
>> Since PEP 517 breaks the current workaround for the caching scheme
>> being inaccurate, the most suitable response is to instead fix pip's
>> caching scheme to use a two tier local cache:
>
>
> I'm still confused -- if setuptools ( invoked  by pip) is producing
> incorrectly named wheels -- surely that's a bug-fix/workaround that should
> go into setuptools?
>
> If the build is being run by pip, then doesn't setuptools have all the
> info about the system that pip has?
>
>
>
>
> Someone building a wheel for distribution is likely intimately aware of
> that project, and can take care to ensure that the wheel is built in such a
> way that it is giving people the most optimal behavior. Pip is auto
> building wheels without human intervention, and as such there is nobody
> there to make sure that we’re not accidentally creating a too-broad wheel,
> so we want to ensure that we have some mechanism in place for not re-using
> the wheel across boundaries that might cause issues.
>
>
>
> we also have plenty of PyPI users that
>> explicitly *opt out* of using publisher-provided pre-built binaries.
>> While Linux distributions are the most common example (see [1] for
>> Fedora's policy, for example), we're not the only ones that have that
>> kind of rule in place.
>
>
> But this is an argument for why pypi should host sdists, and the build
> tools should build sdists, but not why pip should auto-build them.
>
>>
> Condo-forge, for example, almost always builds from source -- sometimes an
> sdist
> from pypi, sometimes a source distribution from github or wherever the
> package is hosted. And sometimes from a git tag ( last resort).
>
>
>
> Pip supports more systems than Conda does, and we do that by relying on
> auto building support. Pip supports systems that don’t have a wheel
> compatibility tag defined for them, and for which we’re unlikely to ever
> have any wheels published for (much less wide spread). It’s pretty easy to
> cover Windows/macOS/Some Linux systems, but when you start talking about
> FreeBSD, OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets
> extremely long.
>
> Pip works in all these situations, and it does so by relying on building
> from source.
>
>
>
> Do the Linux distros use pip to build their packages?
>
>
>
> Not that I am aware of.
>
>
> I tried to do that with conda-packages, and failed due to pip's caching
>  behavior-- it probably would have worked fine in production, but when I
> was developing the build script, I couldn't reliably get pip to ignore
> cached wheels from previous experimental builds.
>
>
>
> Adding —no-cache-dir disables all of pip’s caching.
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Sep 1, 2017, at 2:30 PM, Chris Barker  wrote:
> 
> On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan  > wrote:
> 
> since it
> doesn't reliably distinguish between "this cached wheel was downloaded
> from a repository" and "this wheel was generated locally with a
> particular version of Python".
> 
> It shouldn't have to. sigh.
> 
>  PEP 517 deliberately doesn't let
> frontends do that as part of the initial build process (instead, if
> they want to adjust the tags, they need to do it as a post-processing
> step).
> 
> Since PEP 517 breaks the current workaround for the caching scheme
> being inaccurate, the most suitable response is to instead fix pip's
> caching scheme to use a two tier local cache:
> 
> I'm still confused -- if setuptools ( invoked  by pip) is producing 
> incorrectly named wheels -- surely that's a bug-fix/workaround that should go 
> into setuptools?
> 
> If the build is being run by pip, then doesn't setuptools have all the info 
> about the system that pip has?
> 
> 


Someone building a wheel for distribution is likely intimately aware of that 
project, and can take care to ensure that the wheel is built in such a way that 
it is giving people the most optimal behavior. Pip is auto building wheels 
without human intervention, and as such there is nobody there to make sure that 
we’re not accidentally creating a too-broad wheel, so we want to ensure that we 
have some mechanism in place for not re-using the wheel across boundaries that 
might cause issues.


> 
> we also have plenty of PyPI users that
> explicitly *opt out* of using publisher-provided pre-built binaries.
> While Linux distributions are the most common example (see [1] for
> Fedora's policy, for example), we're not the only ones that have that
> kind of rule in place.
> 
> But this is an argument for why pypi should host sdists, and the build tools 
> should build sdists, but not why pip should auto-build them.
> 
> Condo-forge, for example, almost always builds from source -- sometimes an 
> sdist 
> from pypi, sometimes a source distribution from github or wherever the 
> package is hosted. And sometimes from a git tag ( last resort).


Pip supports more systems than Conda does, and we do that by relying on auto 
building support. Pip supports systems that don’t have a wheel compatibility 
tag defined for them, and for which we’re unlikely to ever have any wheels 
published for (much less wide spread). It’s pretty easy to cover 
Windows/macOS/Some Linux systems, but when you start talking about FreeBSD, 
OpenBSD, Solaris, AIX, HP-UX, etc then the long tail gets extremely long.

Pip works in all these situations, and it does so by relying on building from 
source.


> 
> Do the Linux distros use pip to build their packages?


Not that I am aware of.

> 
> I tried to do that with conda-packages, and failed due to pip's caching  
> behavior-- it probably would have worked fine in production, but when I was 
> developing the build script, I couldn't reliably get pip to ignore cached 
> wheels from previous experimental builds.


Adding —no-cache-dir disables all of pip’s caching.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 1 September 2017 at 19:30, Chris Barker  wrote:
> I'm still confused -- if setuptools ( invoked  by pip) is producing
> incorrectly named wheels -- surely that's a bug-fix/workaround that should
> go into setuptools?
>
> If the build is being run by pip, then doesn't setuptools have all the info
> about the system that pip has?

Simple answer - for historical reasons, and because it's essentially
baked into the design of distutils, setuptools cannot guarantee that
wheels created by two different Python interpreters on a given machine
will NOT be:given the same tags and yet not usable by both. The bug
isn't directly in setuptools - it's typically because the project is
doing unsafe things in setup.py. This is known to happen in real life
- pip has had bug reports for it. In practice, it's sufficiently
difficult to debug the situation, and sufficiently hard to get fixes
made in all of the upstream projects that might need fixing, that we
need to do something in pip.

The solution used in pip (which is a heuristic based on the bug
reports we've seen) is to segregate wheels that we build internally by
the Python interpreter that was used in the build. We don't do that
for downloaded wheels, as we trust that they have been packaged
correctly, with suitable tags.

This sort of "it's really a problem further up the chain, but pip
can't afford to assume it'll get fixed there" problem is typical of
the reason the pp developers have been uncomfortable with the "trust
the backend" approach in the PEP - but they are very much about legacy
issues with the design of distutils that setuptools inherits, and I
imagine that any new build systems and backends will tend to avoid
such problems in their design. So yes, to address another
characterisation of this issue that was mentioned, it's a workaround
for legacy issues. But that legacy will be with us for a long time to
come, in practice.

Sorry - history lesson over ;-)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Fri, Sep 1, 2017 at 11:30 AM, Chris Barker  wrote:
> I'm still confused -- if setuptools ( invoked  by pip) is producing
> incorrectly named wheels -- surely that's a bug-fix/workaround that should
> go into setuptools?
>
> If the build is being run by pip, then doesn't setuptools have all the info
> about the system that pip has?

Some setup.py files are written by project authors who want to use
them to generate wheels for uploading to PyPI, so they're carefully
written to generate good wheels, the wheels go through QA, etc. They
rely on setuptools's current (and fairly sensible) defaults for how to
tag wheels, and if those go wrong, then they take the responsibility
for fixing things.

Other setup.py files were written by project authors who never
considered any possibility outside of 'setup.py install', and haven't
changed since. For them, setuptools's defaults are not so great, but
the authors don't care, because they never guaranteed that it would
work.

Setuptools can't tell which kind of setup.py is calling it. But pip
can make a pretty good guess: if it found a wheel on pypi, then
someone had to have uploaded it, and it's that person's job to make
sure the tags are correct. OTOH if it's taking some random setup.py
file it found in an sdist, then it could be the second type, so better
to play it safe and use a more restrictive wheel tag.

It's a bit quirky and annoying, but that's life. And in the grand
scheme of things this isn't a big deal. The only program that has to
care about this is pip, and pip can always change to a different
heuristic if the situation changes.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

I don't think this is a big deal. Compatibility is a guess, wheel tags do
not and are not intended to guarantee compatibility especially when the
wheel was built locally for caching purposes and not intentionally
published. Anything can happen during a build. PEP 517 won't change this.

So right now pip has an odd way of segregating its wheel cache per
interpreter that involves obscure setup.py arguments. That can be fixed
without re-tagging or asking the build system to re-tag wheels or otherwise
interacting with the new PEP.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 9:23 PM Nick Coghlan  wrote:

since it
> doesn't reliably distinguish between "this cached wheel was downloaded
> from a repository" and "this wheel was generated locally with a
> particular version of Python".


It shouldn't have to. sigh.

>
 PEP 517 deliberately doesn't let
> frontends do that as part of the initial build process (instead, if
> they want to adjust the tags, they need to do it as a post-processing
> step).
>
> Since PEP 517 breaks the current workaround for the caching scheme
> being inaccurate, the most suitable response is to instead fix pip's
> caching scheme to use a two tier local cache:


I'm still confused -- if setuptools ( invoked  by pip) is producing
incorrectly named wheels -- surely that's a bug-fix/workaround that should
go into setuptools?

If the build is being run by pip, then doesn't setuptools have all the info
about the system that pip has?

I'm trying to think through the use case presented the other day:

A package can be build either for all python's or for cpython in
particular. That selection is (currently) done at build time, by custom
code in setup.py.

In this case, the build system SHOULD be able to build the proper wheel,
and label it correctly.  But there might need to be a way to tell the build
system which you want. That logic isn't in existing setup.py files, but
there is only so much you can do!

Then there is the caching problem. If pip finds a pure-python wheel that
matches the spec in the cache, it won't know to go looking for a cpython
specific one. I can't see how separating the downloaded  cache from the
locally built cache will solve that problem:

If a pure-python wheel was already downloaded, then that'll be in the
downloaded cache anyway.

I think the core problem here is having, for instance, pypy and cpython
sharing a wheel cache. Is it really that important to support that?

If so, this is a total kludge, but maybe the pip cache could keep a
record-- I looked for a cpython specific wheel, and it doesn't exist, so I
never need to look again.

Or -- maybe more kludgy -- every time pip puts a  wheel in the cache via
cpython, it makes a link with the cpython name.

Independently of that though, it's also important to note that pip
> probably *won't* switch to invoking legacy setup.py files through the
> PEP 517 backend - instead, it will only do that if there's a
> pyproject.toml file that says "Use the setuptools PEP 517 backend".


That's probably a good way to keep backward compatibility while moving
forward. Which means pip can have two modes -- legacy mode and pep517 mode.

Which may encourage me to stop being such a noodge -- As long as we're
talking legacy mode, we can be as hacky as we need to be.

Autobuilding support definitely *won't* be removed from pip,


I'm not suggesting it should be (and I think I may be the only one that
might have been) However, I do think auto-building support does not have to
support all the complex use cases.

we also have plenty of PyPI users that
> explicitly *opt out* of using publisher-provided pre-built binaries.
> While Linux distributions are the most common example (see [1] for
> Fedora's policy, for example), we're not the only ones that have that
> kind of rule in place.


But this is an argument for why pypi should host sdists, and the build
tools should build sdists, but not why pip should auto-build them.

>
Condo-forge, for example, almost always builds from source -- sometimes an
sdist
from pypi, sometimes a source distribution from github or wherever the
package is hosted. And sometimes from a git tag ( last resort).

Do the Linux distros use pip to build their packages?

I tried to do that with conda-packages, and failed due to pip's caching
 behavior-- it probably would have worked fine in production, but when I
was developing the build script, I couldn't reliably get pip to ignore
cached wheels from previous experimental builds.

So I ( and the entire community, as far as I can tell) stuck with calling
setuptools directly - via setup.py, but with setuptools-specific flags)

As pip is a package management tool, it makes sense that other package
managers wouldn't need to use it.

NOTE: I do use pip to build conda packages from wheels in a couple cases --
but that's considered a work around for my pathetic build skills -- and
even then, I don't have pip download the wheel -- that's conda-build's job.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Sep 1, 2017, at 12:23 AM, Nick Coghlan  wrote:
> 
> 1. Wheel download cache (if this gets a hit, don't even check the package 
> repo)
> 2. Check the package repo for a wheel file (and if you find one, add
> it to the download cache)
> 3. Local build cache (make this interpreter implementation specific)
> 4. Build a local wheel that's specific to the running interpreter 
> implementation

This is basically what exists now. We just force a compatibility tag to handle 
(4).  
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nick Coghlan]

On 1 September 2017 at 07:29, Chris Barker  wrote:
> But I do think we should be clear about what is a hack for backward
> compatibility, and what is part of the designed functionality.

Right, and I think it's pretty clear that the problem xoviat raised is
with the way pip's implicit local artifact cache works, since it
doesn't reliably distinguish between "this cached wheel was downloaded
from a repository" and "this wheel was generated locally with a
particular version of Python".

Currently pip is working around that problem by overriding the
compatibility tags produced by setup.py to ensure the generated wheel
files are interpreter specific, but PEP 517 deliberately doesn't let
frontends do that as part of the initial build process (instead, if
they want to adjust the tags, they need to do it as a post-processing
step).

Since PEP 517 breaks the current workaround for the caching scheme
being inaccurate, the most suitable response is to instead fix pip's
caching scheme to use a two tier local cache:

1. Wheel download cache (if this gets a hit, don't even check the package repo)
2. Check the package repo for a wheel file (and if you find one, add
it to the download cache)
3. Local build cache (make this interpreter implementation specific)
4. Build a local wheel that's specific to the running interpreter implementation

Independently of that though, it's also important to note that pip
probably *won't* switch to invoking legacy setup.py files through the
PEP 517 backend - instead, it will only do that if there's a
pyproject.toml file that says "Use the setuptools PEP 517 backend". In
such cases, it's reasonable to expect that the project will also be
using environment markers appropriately, and not putting conditional
dependency installation logic directly into setup.py.

Autobuilding support definitely *won't* be removed from pip, as there
will not only always be a large number of projects that don't provide
their own prebuilt wheel files, we also have plenty of PyPI users that
explicitly *opt out* of using publisher-provided pre-built binaries.
While Linux distributions are the most common example (see [1] for
Fedora's policy, for example), we're not the only ones that have that
kind of rule in place.

Cheers,
Nick.

[1] 
https://fedoraproject.org/wiki/Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker - NOAA Federal]

One thing to keep in mind is that there are quite a few projects on pypa
with pure python source distributions uploaded that will not be updated and
people may still desire to use. We want pip to be able to still build and
install them.


That is the challenge!

But the motivating use case here was a package with a c extension that was
optionally built.

That's the rare case.

Regular old pure-python packages are the usual case, and a lot easier.

-CHB

2017-08-31 16:29 GMT-05:00 Chris Barker :

> On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith  wrote:
>
>> > Surely the build system should know how to correctly name the wheel it
>> builds.
>>
>> It's probably worth mentioning the specific problem that motivated pip
>> to start doing this.
>>
>> It used to be standard, and is still quite common, for setup.py
>> scripts to contain stuff like:
>>
>> install_requires = [...]
>> if sys.version_info < (3, 4):
>> install_requires += [...]
>> if platform.python_implementation() == "PyPy":
>> install_requires += [...]
>>
>> setup(..., install_requires=install_requires)
>>
>> This kind of logic in setup.py worked fine in the old days when all
>> you did was 'setup.py install', but then wheels came along
>
>
> And indeed, setuptools originally used easy_install, which was part of
> setuptools...
>
> and
>> retroactively turned lots of setup.py scripts from working into
>> broken. The problem is that with this kind of setup.py,
>>
>
>
>> But it will take a while for existing setup.py files transition to
>> using those, and in the mean time pip can't assume that a random wheel
>> generated by 'setup.py bdist_wheel' has accurate Python tags.
>>
>
> This was my original point -- I understand that we want "pip install" to
> continue to work for, hopefully, everything it works for now.
>
> But I do think we should be clear about what is a hack for backward
> compatibility, and what is part of the designed functionality.
>
> Sorry to be poking at all this from the fringes (Not having been all that
> involved in a very long discussion), it's just that the whole
>
> distutils--setuptools--pip--distribute--setuptools--pip
>
> stack has a LOT of legacy cruft, and I'm  concerned that the efforts for
> backward compatibility may end up leading us to another poorly de-coupled
> design.
>
>
>> Hopefully new legacy-free backends will get this right from the start.
>>
>
> exactly -- let's keep the "backward compatibility hack" labels clear!
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

I don't buy the argument that pip needs to keep separate caches for
different implementations. According to the wheel specification, a wheel is
valid for the Python implementations that it specifies. This is therefore a
blatant violation by setuptools of an existing specification that, in
theory, should be ignored by pip.

That said, setuptools is currently the de facto build system, meaning that
we cannot simply break the status quo. So here's my proposal:

1. For now, wheels are renamed if and only if the backend is setuptools.
2. In the future, we introduce a new setup() function that is imported from
a different module path that requires compliance with the specification.
3. Once the vast majority of projects are using the new setup function, we
can adjust the pep 517 setuptools build backend to use the most restrictive
tag for the old function
4. We can finally remove autobuilding from pip.

One thing to keep in mind is that there are quite a few projects on pypa
with pure python source distributions uploaded that will not be updated and
people may still desire to use. We want pip to be able to still build and
install them.

2017-08-31 16:29 GMT-05:00 Chris Barker :

> On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith  wrote:
>
>> > Surely the build system should know how to correctly name the wheel it
>> builds.
>>
>> It's probably worth mentioning the specific problem that motivated pip
>> to start doing this.
>>
>> It used to be standard, and is still quite common, for setup.py
>> scripts to contain stuff like:
>>
>> install_requires = [...]
>> if sys.version_info < (3, 4):
>> install_requires += [...]
>> if platform.python_implementation() == "PyPy":
>> install_requires += [...]
>>
>> setup(..., install_requires=install_requires)
>>
>> This kind of logic in setup.py worked fine in the old days when all
>> you did was 'setup.py install', but then wheels came along
>
>
> And indeed, setuptools originally used easy_install, which was part of
> setuptools...
>
> and
>> retroactively turned lots of setup.py scripts from working into
>> broken. The problem is that with this kind of setup.py,
>>
>
>
>> But it will take a while for existing setup.py files transition to
>> using those, and in the mean time pip can't assume that a random wheel
>> generated by 'setup.py bdist_wheel' has accurate Python tags.
>>
>
> This was my original point -- I understand that we want "pip install" to
> continue to work for, hopefully, everything it works for now.
>
> But I do think we should be clear about what is a hack for backward
> compatibility, and what is part of the designed functionality.
>
> Sorry to be poking at all this from the fringes (Not having been all that
> involved in a very long discussion), it's just that the whole
>
> distutils--setuptools--pip--distribute--setuptools--pip
>
> stack has a LOT of legacy cruft, and I'm  concerned that the efforts for
> backward compatibility may end up leading us to another poorly de-coupled
> design.
>
>
>> Hopefully new legacy-free backends will get this right from the start.
>>
>
> exactly -- let's keep the "backward compatibility hack" labels clear!
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith  wrote:

> > Surely the build system should know how to correctly name the wheel it
> builds.
>
> It's probably worth mentioning the specific problem that motivated pip
> to start doing this.
>
> It used to be standard, and is still quite common, for setup.py
> scripts to contain stuff like:
>
> install_requires = [...]
> if sys.version_info < (3, 4):
> install_requires += [...]
> if platform.python_implementation() == "PyPy":
> install_requires += [...]
>
> setup(..., install_requires=install_requires)
>
> This kind of logic in setup.py worked fine in the old days when all
> you did was 'setup.py install', but then wheels came along


And indeed, setuptools originally used easy_install, which was part of
setuptools...

and
> retroactively turned lots of setup.py scripts from working into
> broken. The problem is that with this kind of setup.py,
>


> But it will take a while for existing setup.py files transition to
> using those, and in the mean time pip can't assume that a random wheel
> generated by 'setup.py bdist_wheel' has accurate Python tags.
>

This was my original point -- I understand that we want "pip install" to
continue to work for, hopefully, everything it works for now.

But I do think we should be clear about what is a hack for backward
compatibility, and what is part of the designed functionality.

Sorry to be poking at all this from the fringes (Not having been all that
involved in a very long discussion), it's just that the whole

distutils--setuptools--pip--distribute--setuptools--pip

stack has a LOT of legacy cruft, and I'm  concerned that the efforts for
backward compatibility may end up leading us to another poorly de-coupled
design.


> Hopefully new legacy-free backends will get this right from the start.
>

exactly -- let's keep the "backward compatibility hack" labels clear!

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

There are two points that I would note here:
1. setuptools handling of environment markers in wheels was actually broken
until recently.
2. Even now, people still prefer runtime detection of environment markers
to work around pip's upgrade behavior and old setuptools versions. There
needs to be a public effort to change this behavior.

For #2, it's really not a clean solution, but it would be more effective to
scan the bytecode of setup.py and then warn users that this behavior is
deprecated. I expect that you would get a lot of complaints from package
authors but that is the point.

2017-08-31 13:03 GMT-05:00 Nathaniel Smith :

> On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal
>  wrote:
> > The package manager should manage the package, not built it, or change
> it.
> >
> > Surely the build system should know how to correctly name the wheel it
> builds.
>
> It's probably worth mentioning the specific problem that motivated pip
> to start doing this.
>
> It used to be standard, and is still quite common, for setup.py
> scripts to contain stuff like:
>
> install_requires = [...]
> if sys.version_info < (3, 4):
> install_requires += [...]
> if platform.python_implementation() == "PyPy":
> install_requires += [...]
>
> setup(..., install_requires=install_requires)
>
> This kind of logic in setup.py worked fine in the old days when all
> you did was 'setup.py install', but then wheels came along and
> retroactively turned lots of setup.py scripts from working into
> broken. The problem is that with this kind of setup.py, setuptools has
> *no idea* that the install_requires you gave it would have been
> different if you had run setup.py with a different version of Python,
> so when it has to assign Python tags to a built wheel it guesses wrong
> and uses ones that are too general.
>
> The right way to do this is to use PEP 508 environment markers:
> https://www.python.org/dev/peps/pep-0508/#environment-markers
> or the non-standard extras hack:
> https://wheel.readthedocs.io/en/latest/#defining-
> conditional-dependencies
> Both of these let you export the whole requirements-choosing logic
> into the wheel metadata, so that it can be evaluated at install time
> instead of build time.
>
> But it will take a while for existing setup.py files transition to
> using those, and in the mean time pip can't assume that a random wheel
> generated by 'setup.py bdist_wheel' has accurate Python tags.
>
> Hopefully new legacy-free backends will get this right from the start.
> For example flit makes it impossible to get this wrong.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

Don't be afraid of extras. They have not been PEP'd but are required for a
useful Python packaging system. install_requires is just a shorthand for
extras with the "" empty string as their name. Too bad we couldn't agree on
; vs : as the delimiter though.

On Thu, Aug 31, 2017 at 2:04 PM Nathaniel Smith  wrote:

> On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal
>  wrote:
> > The package manager should manage the package, not built it, or change
> it.
> >
> > Surely the build system should know how to correctly name the wheel it
> builds.
>
> It's probably worth mentioning the specific problem that motivated pip
> to start doing this.
>
> It used to be standard, and is still quite common, for setup.py
> scripts to contain stuff like:
>
> install_requires = [...]
> if sys.version_info < (3, 4):
> install_requires += [...]
> if platform.python_implementation() == "PyPy":
> install_requires += [...]
>
> setup(..., install_requires=install_requires)
>
> This kind of logic in setup.py worked fine in the old days when all
> you did was 'setup.py install', but then wheels came along and
> retroactively turned lots of setup.py scripts from working into
> broken. The problem is that with this kind of setup.py, setuptools has
> *no idea* that the install_requires you gave it would have been
> different if you had run setup.py with a different version of Python,
> so when it has to assign Python tags to a built wheel it guesses wrong
> and uses ones that are too general.
>
> The right way to do this is to use PEP 508 environment markers:
> https://www.python.org/dev/peps/pep-0508/#environment-markers
> or the non-standard extras hack:
>
> https://wheel.readthedocs.io/en/latest/#defining-conditional-dependencies
> Both of these let you export the whole requirements-choosing logic
> into the wheel metadata, so that it can be evaluated at install time
> instead of build time.
>
> But it will take a while for existing setup.py files transition to
> using those, and in the mean time pip can't assume that a random wheel
> generated by 'setup.py bdist_wheel' has accurate Python tags.
>
> Hopefully new legacy-free backends will get this right from the start.
> For example flit makes it impossible to get this wrong.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Thu, Aug 31, 2017 at 8:41 AM, Chris Barker - NOAA Federal
 wrote:
> The package manager should manage the package, not built it, or change it.
>
> Surely the build system should know how to correctly name the wheel it builds.

It's probably worth mentioning the specific problem that motivated pip
to start doing this.

It used to be standard, and is still quite common, for setup.py
scripts to contain stuff like:

install_requires = [...]
if sys.version_info < (3, 4):
install_requires += [...]
if platform.python_implementation() == "PyPy":
install_requires += [...]

setup(..., install_requires=install_requires)

This kind of logic in setup.py worked fine in the old days when all
you did was 'setup.py install', but then wheels came along and
retroactively turned lots of setup.py scripts from working into
broken. The problem is that with this kind of setup.py, setuptools has
*no idea* that the install_requires you gave it would have been
different if you had run setup.py with a different version of Python,
so when it has to assign Python tags to a built wheel it guesses wrong
and uses ones that are too general.

The right way to do this is to use PEP 508 environment markers:
https://www.python.org/dev/peps/pep-0508/#environment-markers
or the non-standard extras hack:
https://wheel.readthedocs.io/en/latest/#defining-conditional-dependencies
Both of these let you export the whole requirements-choosing logic
into the wheel metadata, so that it can be evaluated at install time
instead of build time.

But it will take a while for existing setup.py files transition to
using those, and in the mean time pip can't assume that a random wheel
generated by 'setup.py bdist_wheel' has accurate Python tags.

Hopefully new legacy-free backends will get this right from the start.
For example flit makes it impossible to get this wrong.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

On Thu, Aug 31, 2017 at 12:58 PM Chris Barker  wrote:

> On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore  wrote:
>
>> > As to using pip to build wheels -- there is good reason to do that
>> > now, but in s post PEP 517 world, one would call the build system
>> > directly to build a wheel-- after all, all pip should be  doing is
>> > calling the build system anyway.
>>
>> I disagree - "pip wheel" will still be useful, for example to obtain a
>> wheel from PyPI either by downloading or download & build. Also just
>> to have a unified interface that works regardless of the project
>> backend - if a project switches from setuptools to flit, for example,
>> it would be good if deployment and test scripts didn't have to change.
>>
>
> Isn't that why we have PEP 517? I unified interface to build tools?
>
> So I still expect pip wheel to be useful in a post-PEP 517 world.
>
>
> Maybe so -- but all pip should be doing is passing off the job to the
> back-end.
>
> Again, the package manager, well, manages the packages. It shouldn't be
> concerned with how to build them.
>

pip has always been under appreciated as a build tool. Suppose the next
version of pip dropped support for sdists and could only install wheels.
Then it would not be a build tool. Very little changes when it switches to
supporting sdists from its own setup.py machinery to a more pluggable
mechanism. It is a little controversial that it can produce wheels for
re-use but the feature is very practical. You could put 'pip wheel' under a
different command line tool but why bother.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 31 August 2017 at 17:50, Chris Barker  wrote:
>> So I still expect pip wheel to be useful in a post-PEP 517 world.
>
> Maybe so -- but all pip should be doing is passing off the job to the
> back-end.

It also handles locating the appropriate version on PyPI, downloading
(including caching the downloads), building via sdist to avoid
discrepancies caused by out of date intermediates (yes, the backend
should be doing this, let's not open up the "trust the backend"
argument again ;-)) Pip also handles *installing* the appropriate
backends into the build environment.

Certainly "pip install ." is mostly redundant compared to "flit build"
(although again, pip provides a unified command line interface - you
*could* read pyproject.toml in your deployment script, then install
the backend and call the build_wheel hook, but why do that rather than
using at least *some* frontend, if not pip?

> Again, the package manager, well, manages the packages. It shouldn't be
> concerned with how to build them.

It isn't - it just manages organising the process of getting the build
tools and running them.

A simpler competitor to pip that *only* works with wheels and knows
nothing about the build process is entirely possible under PEP 517.
Indeed such a tool probably doesn't even need PEP 517. But there
hasn't been enough demand for such a tool to be built yet, so I doubt
people will suddenly be crying out for one when PEP 517 is
implemented...

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Aug 31, 2017, at 12:58 PM, Chris Barker  wrote:
> 
> In fact, I see PEP 517 as a step toward more wheels, and fewer sdists for 
> distribution.


Everything should ideally still have a sdist.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 9:04 AM, Donald Stufft  wrote:

> I'm a bit confused -- are we talking about the backwards compatible
> path to the future -- or the end-game?
>
>
> Pip purposely overrides what setuptools tags the wheel with in order to
> make it extremely specific to the Python it is currently being run on. Now,
> a big part of why it needs to do this, is because a lot of setup.py were
> not written with wheel in mind, so they produce broken wheels by default.
>

right -- just so we're clear that this is hack to maintain backward
compatibility. Though that seems like a hack that setuptools should have,
not pip.


> One such example is a setup.py that produces a pure Python wheel for PyPy,
> and a C-extension wheel for Python. If someone installs that project on
> PyPy first and they only have a sdist available on PyPI, then pip will
> cache a pure Python wheel for that project, and by default (without this
> bit to force it) that wheel would be acceptable for CPython too and would
> have been used instead of building a wheel with a compiled extension for
> CPython.
>

Frankly, I'm not sure that pip should be sharing a cache between PyPy and
cPython anyway..

(I've had other issues with pip's fairly aggressive caching in the past...)

But if it does, then pip should always look to see if there is a more
"specific" version of a package available anyway.

But I see your point -- if there is a cached pure-python wheel, pip can go
look on PyPi for a cpython wheel, but if it isn't there, and there is only
an sdist, it would have to download and build it to find out if there is a
cPython version. Which of course, would then happen (and be wasted effort)
 essentially all the time, as most packages are pure python.

But the proper solution to this is for packages that have both a pure
python and cpython version available to put the wheels up on pypi. After
all, this is the unusual case.

In fact, I see PEP 517 as a step toward more wheels, and fewer sdists for
distribution.

-CHB











> You could argue that the new crop of build backends should handle this
> better— and hopefully you’re correct— but I think that the two desires are
> at odd with one another. Most build backends are going to want to, by
> default, tag the wheel with as general of a tag as they think will work
> given the information available to them, whereas with pip’s internal wheel
> cache, we generally want that wheel to be as specific to the current
> version of Python as possible.
>
> Another possible solution is to stop using the wheel tagging format to
> encode this information for pip’s internal wheel cache and to encode it
> into the path, so that instead of having PyPy and CPython share a cache
> directory, they each have their own. That isn’t an unreasonable mechanism
> for doing that and which one gets used is an implementation detail of pip.
>
> —
> Donald Stufft
>
>
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore  wrote:

> > As to using pip to build wheels -- there is good reason to do that
> > now, but in s post PEP 517 world, one would call the build system
> > directly to build a wheel-- after all, all pip should be  doing is
> > calling the build system anyway.
>
> I disagree - "pip wheel" will still be useful, for example to obtain a
> wheel from PyPI either by downloading or download & build. Also just
> to have a unified interface that works regardless of the project
> backend - if a project switches from setuptools to flit, for example,
> it would be good if deployment and test scripts didn't have to change.
>

Isn't that why we have PEP 517? I unified interface to build tools?

So I still expect pip wheel to be useful in a post-PEP 517 world.


Maybe so -- but all pip should be doing is passing off the job to the
back-end.

Again, the package manager, well, manages the packages. It shouldn't be
concerned with how to build them.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

I'm not against renaming a wheel to adjust the tags, but it is a strange
caching strategy. wheel is due for a 'retag' feature that does a more
rigorous job of changing tags.

The build system should name its wheels correctly. Normally the pyNN tag
would be used for a pure distribution and the cpNN tag when there are
extensions. If you want to affect the tagging when called through PEP 517,
edit the source's pyproject.toml / setup.cfg / setup.py.

It will probably always make sense to use pip to build wheels when you want
to build wheels for a project and all its dependencies. Then you can keep
these wheels in a folder for repeatable offline deployment.

On Thu, Aug 31, 2017 at 11:48 AM Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:

> > that neither pip nor the setuptools backend should not change the tags
> > it applies to wheels by default).
>
> I'm a bit confused -- are we talking about the backwards compatible
> path to the future -- or the end-game?
>
> In short -- I'm sure we'll have to do some hacky stuff to keep
> backwards compatibility, but the end game should be a clean separation
> of concerns :
>
> Pip (or any front end) should never "build a wheel", and it certainly
> shouldn't have to know enough about what's in a wheel to be re-naming
> it for generic python vs cpython.
>
> The package manager should manage the package, not built it, or change it.
>
> Surely the build system should know how to correctly name the wheel it
> builds.
>
> As to using pip to build wheels -- there is good reason to do that
> now, but in s post PEP 517 world, one would call the build system
> directly to build a wheel-- after all, all pip should be  doing is
> calling the build system anyway.
>
> -CHB
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Aug 31, 2017, at 11:41 AM, Chris Barker - NOAA Federal 
>  wrote:
> 
>> that neither pip nor the setuptools backend should not change the tags
>> it applies to wheels by default).
> 
> I'm a bit confused -- are we talking about the backwards compatible
> path to the future -- or the end-game?
> 
> In short -- I'm sure we'll have to do some hacky stuff to keep
> backwards compatibility, but the end game should be a clean separation
> of concerns :
> 
> Pip (or any front end) should never "build a wheel", and it certainly
> shouldn't have to know enough about what's in a wheel to be re-naming
> it for generic python vs cpython.
> 
> The package manager should manage the package, not built it, or change it.
> 
> Surely the build system should know how to correctly name the wheel it builds.
> 
> As to using pip to build wheels -- there is good reason to do that
> now, but in s post PEP 517 world, one would call the build system
> directly to build a wheel-- after all, all pip should be  doing is
> calling the build system anyway.
> 


Pip purposely overrides what setuptools tags the wheel with in order to make it 
extremely specific to the Python it is currently being run on. Now, a big part 
of why it needs to do this, is because a lot of setup.py were not written with 
wheel in mind, so they produce broken wheels by default.

One such example is a setup.py that produces a pure Python wheel for PyPy, and 
a C-extension wheel for Python. If someone installs that project on PyPy first 
and they only have a sdist available on PyPI, then pip will cache a pure Python 
wheel for that project, and by default (without this bit to force it) that 
wheel would be acceptable for CPython too and would have been used instead of 
building a wheel with a compiled extension for CPython.

You could argue that the new crop of build backends should handle this better— 
and hopefully you’re correct— but I think that the two desires are at odd with 
one another. Most build backends are going to want to, by default, tag the 
wheel with as general of a tag as they think will work given the information 
available to them, whereas with pip’s internal wheel cache, we generally want 
that wheel to be as specific to the current version of Python as possible.

Another possible solution is to stop using the wheel tagging format to encode 
this information for pip’s internal wheel cache and to encode it into the path, 
so that instead of having PyPy and CPython share a cache directory, they each 
have their own. That isn’t an unreasonable mechanism for doing that and which 
one gets used is an implementation detail of pip.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 31 August 2017 at 16:41, Chris Barker - NOAA Federal
 wrote:
> As to using pip to build wheels -- there is good reason to do that
> now, but in s post PEP 517 world, one would call the build system
> directly to build a wheel-- after all, all pip should be  doing is
> calling the build system anyway.

I disagree - "pip wheel" will still be useful, for example to obtain a
wheel from PyPI either by downloading or download & build. Also just
to have a unified interface that works regardless of the project
backend - if a project switches from setuptools to flit, for example,
it would be good if deployment and test scripts didn't have to change.

So I still expect pip wheel to be useful in a post-PEP 517 world.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker - NOAA Federal]

> that neither pip nor the setuptools backend should not change the tags
> it applies to wheels by default).

I'm a bit confused -- are we talking about the backwards compatible
path to the future -- or the end-game?

In short -- I'm sure we'll have to do some hacky stuff to keep
backwards compatibility, but the end game should be a clean separation
of concerns :

Pip (or any front end) should never "build a wheel", and it certainly
shouldn't have to know enough about what's in a wheel to be re-naming
it for generic python vs cpython.

The package manager should manage the package, not built it, or change it.

Surely the build system should know how to correctly name the wheel it builds.

As to using pip to build wheels -- there is good reason to do that
now, but in s post PEP 517 world, one would call the build system
directly to build a wheel-- after all, all pip should be  doing is
calling the build system anyway.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 31 August 2017 at 06:36, Nathaniel Smith  wrote:
> On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan  wrote:
>> On 31 August 2017 at 14:22, xoviat  wrote:
>>> Again, let me repeat that: wheels generated using setuptools are valid for
>>> CPython only if build on CPython. This is not the current setuptools
>>> behavior but will be for all setuptools build backend calls (I assume legacy
>>> will remain the same).
>>
>> While I do think your proposal would work (on the assumption that
>> folks aren't use "pip wheel" to generate their wheel files for
>> upload),
>
> I use 'pip wheel' to generate wheel files for upload... (I like to
> generate an sdist and then build a wheel from that, and 'pip wheel
> sdist.tar.gz' is more convenient than manually unpacking and running
> bdist_wheel. )

Agreed. The wheels built by "pip wheel" mustn't change (and that means
that neither pip nor the setuptools backend should not change the tags
it applies to wheels by default).

The fact that pip artificially restricts wheels it builds as part of
its internal use to be platform-specific is a workaround for an
underlying problem, which is that it cannot guarantee that an
internally-built wheel will be the same as one that the project later
publishes for that version (it's a variation on the wheel equivalency
problem we've been discussing). Rather than artificially modifying the
compatibility tags, pip should cache internally-built wheels
differently, so that wheels published by the upstream project have
priority over pip-built wheels. That's a change that purely affects
pip and is unrelated to PEP 517 directly, but it's something pip
probably needs to do to prepare for switching to PEP 517. (xoviat -
sorry I didn't think of this when commenting on the PR - it was Nick's
suggestion here that made the situation clearer to me).

>> an alternative approach with a lower risk of unintended side
>> effects would be for *pip* to either rename the autobuilt file before
>> adding it to the cache, or else to adjust its caching strategy a bit
>> to internally separate a shared wheel download cache from
>> per-interpreter-compatibility-tag caches for locally built wheel
>> files.
>
> +1

Yes. Renaming is a hack, to be honest, and improving the caching
strategy would be my preference.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Will do. Better to hash this out sooner than later.

On Aug 31, 2017 12:36 AM, "Nathaniel Smith"  wrote:

> On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan  wrote:
> > On 31 August 2017 at 14:22, xoviat  wrote:
> >> Again, let me repeat that: wheels generated using setuptools are valid
> for
> >> CPython only if build on CPython. This is not the current setuptools
> >> behavior but will be for all setuptools build backend calls (I assume
> legacy
> >> will remain the same).
> >
> > While I do think your proposal would work (on the assumption that
> > folks aren't use "pip wheel" to generate their wheel files for
> > upload),
>
> I use 'pip wheel' to generate wheel files for upload... (I like to
> generate an sdist and then build a wheel from that, and 'pip wheel
> sdist.tar.gz' is more convenient than manually unpacking and running
> bdist_wheel. )
>
> > an alternative approach with a lower risk of unintended side
> > effects would be for *pip* to either rename the autobuilt file before
> > adding it to the cache, or else to adjust its caching strategy a bit
> > to internally separate a shared wheel download cache from
> > per-interpreter-compatibility-tag caches for locally built wheel
> > files.
>
> +1
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Wed, Aug 30, 2017 at 9:56 PM, Nick Coghlan  wrote:
> On 31 August 2017 at 14:22, xoviat  wrote:
>> Again, let me repeat that: wheels generated using setuptools are valid for
>> CPython only if build on CPython. This is not the current setuptools
>> behavior but will be for all setuptools build backend calls (I assume legacy
>> will remain the same).
>
> While I do think your proposal would work (on the assumption that
> folks aren't use "pip wheel" to generate their wheel files for
> upload),

I use 'pip wheel' to generate wheel files for upload... (I like to
generate an sdist and then build a wheel from that, and 'pip wheel
sdist.tar.gz' is more convenient than manually unpacking and running
bdist_wheel. )

> an alternative approach with a lower risk of unintended side
> effects would be for *pip* to either rename the autobuilt file before
> adding it to the cache, or else to adjust its caching strategy a bit
> to internally separate a shared wheel download cache from
> per-interpreter-compatibility-tag caches for locally built wheel
> files.

+1

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nick Coghlan]

On 31 August 2017 at 14:22, xoviat  wrote:
> This appears to be related to https://github.com/pypa/pip/pull/3225, where
> pip would generate overbroad wheel names in the cache. So to proceed on this
> issue, setuptools will assume that generated wheels are valid for the
> specific python implementation only (eg. wheels generated for CPython will
> not be valid for pypy).
>
> Again, let me repeat that: wheels generated using setuptools are valid for
> CPython only if build on CPython. This is not the current setuptools
> behavior but will be for all setuptools build backend calls (I assume legacy
> will remain the same).

While I do think your proposal would work (on the assumption that
folks aren't use "pip wheel" to generate their wheel files for
upload), an alternative approach with a lower risk of unintended side
effects would be for *pip* to either rename the autobuilt file before
adding it to the cache, or else to adjust its caching strategy a bit
to internally separate a shared wheel download cache from
per-interpreter-compatibility-tag caches for locally built wheel
files.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

This appears to be related to https://github.com/pypa/pip/pull/3225, where
pip would generate overbroad wheel names in the cache. So to proceed on
this issue, setuptools will assume that generated wheels are valid for the
specific python implementation only (eg. wheels generated for CPython will
not be valid for pypy).

Again, let me repeat that: wheels generated using setuptools are valid for
CPython only if build on CPython. This is not the current setuptools
behavior but will be for all setuptools build backend calls (I assume
legacy will remain the same).

Jacob, Paul, Donald, is everyone okay with that? And does everyone
understand the reason for this?

2017-08-30 18:29 GMT-05:00 xoviat :

> So while I was working on pip, I noticed a parameter called 'autobuilding'
> in wheel.py. What this does (I think) is that if pip decides that it needs
> to be enabled (which is an implementation detail not relevant here), it
> adds the following to the "setup.py bdist_wheel" call:
>
> ["--python-tag", python_tag]
>
> This switches the output of the bdist_wheel filename to include something
> like "cp36" rather than "py36," which indicates the Python implementation.
> The question is this: this behavior isn't captured (I think) in the
> build_wheel function, possibly (probably) for good reason. So that means
> once everything goes through the PEP 517 protocol, pip will not be able to
> let setuptools know about this information. Which means that we need to
> decide:
>
> Should the PEP 517 backend for setuptools assume "cp36" in python_tag
> OR
> Should the PEP 517 backend for setuptools assume "py36" in python_tag
>
>
> 2017-08-29 18:51 GMT-05:00 Matthew Brett :
>
>> Hi,
>>
>> On Tue, Aug 29, 2017 at 8:30 PM, xoviat  wrote:
>> >> For me, using NotImplemented is a misuse of the singleton since I know
>> >> what it's meant to be used for (and so I cringe every time I hear it
>> brought
>> >> up as a solution).
>> >
>> > I hate to reiterate too much, but I think that you are correct here:
>> > NotImplemented is not Pythonic. It's a complete hack that I would think
>> > people would like to quarantine to the limited use case of binary
>> > operations. Even with the poorly contextualized email that Guido
>> received, I
>> > think it was clear that he thought it wasn't appropriate outside of
>> that use
>> > case either (I don't think better context would have changed that).
>> >
>> > Personally, I was willing to overlook that for the sake of a speedy
>> > resolution. But reading the email chain, it seems that only Thomas and
>> > Daniel are currently in favor of using NotImplemented, which is a shift
>> from
>> > the earlier situation. So now it seems that NotImplemented is not
>> likely to
>> > be used.
>>
>> Just for the record, I thought Nathaniel's NotImplemented suggestion
>> was a good one, and his explanation of why, was clear and convincing,
>> but he's already accepted the alternative, of a custom error, so I
>> don't think there's much point on going back over that.
>>
>> Also just for the record, I'd like to warmly thank y'all for your
>> great patience in this long discussion.
>>
>> Cheers,
>>
>> Matthew
>>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

So while I was working on pip, I noticed a parameter called 'autobuilding'
in wheel.py. What this does (I think) is that if pip decides that it needs
to be enabled (which is an implementation detail not relevant here), it
adds the following to the "setup.py bdist_wheel" call:

["--python-tag", python_tag]

This switches the output of the bdist_wheel filename to include something
like "cp36" rather than "py36," which indicates the Python implementation.
The question is this: this behavior isn't captured (I think) in the
build_wheel function, possibly (probably) for good reason. So that means
once everything goes through the PEP 517 protocol, pip will not be able to
let setuptools know about this information. Which means that we need to
decide:

Should the PEP 517 backend for setuptools assume "cp36" in python_tag
OR
Should the PEP 517 backend for setuptools assume "py36" in python_tag


2017-08-29 18:51 GMT-05:00 Matthew Brett :

> Hi,
>
> On Tue, Aug 29, 2017 at 8:30 PM, xoviat  wrote:
> >> For me, using NotImplemented is a misuse of the singleton since I know
> >> what it's meant to be used for (and so I cringe every time I hear it
> brought
> >> up as a solution).
> >
> > I hate to reiterate too much, but I think that you are correct here:
> > NotImplemented is not Pythonic. It's a complete hack that I would think
> > people would like to quarantine to the limited use case of binary
> > operations. Even with the poorly contextualized email that Guido
> received, I
> > think it was clear that he thought it wasn't appropriate outside of that
> use
> > case either (I don't think better context would have changed that).
> >
> > Personally, I was willing to overlook that for the sake of a speedy
> > resolution. But reading the email chain, it seems that only Thomas and
> > Daniel are currently in favor of using NotImplemented, which is a shift
> from
> > the earlier situation. So now it seems that NotImplemented is not likely
> to
> > be used.
>
> Just for the record, I thought Nathaniel's NotImplemented suggestion
> was a good one, and his explanation of why, was clear and convincing,
> but he's already accepted the alternative, of a custom error, so I
> don't think there's much point on going back over that.
>
> Also just for the record, I'd like to warmly thank y'all for your
> great patience in this long discussion.
>
> Cheers,
>
> Matthew
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Matthew Brett]

Hi,

On Tue, Aug 29, 2017 at 8:30 PM, xoviat  wrote:
>> For me, using NotImplemented is a misuse of the singleton since I know
>> what it's meant to be used for (and so I cringe every time I hear it brought
>> up as a solution).
>
> I hate to reiterate too much, but I think that you are correct here:
> NotImplemented is not Pythonic. It's a complete hack that I would think
> people would like to quarantine to the limited use case of binary
> operations. Even with the poorly contextualized email that Guido received, I
> think it was clear that he thought it wasn't appropriate outside of that use
> case either (I don't think better context would have changed that).
>
> Personally, I was willing to overlook that for the sake of a speedy
> resolution. But reading the email chain, it seems that only Thomas and
> Daniel are currently in favor of using NotImplemented, which is a shift from
> the earlier situation. So now it seems that NotImplemented is not likely to
> be used.

Just for the record, I thought Nathaniel's NotImplemented suggestion
was a good one, and his explanation of why, was clear and convincing,
but he's already accepted the alternative, of a custom error, so I
don't think there's much point on going back over that.

Also just for the record, I'd like to warmly thank y'all for your
great patience in this long discussion.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> For me, using NotImplemented is a misuse of the singleton since I know
what it's meant to be used for (and so I cringe every time I hear it
brought up as a solution).

I hate to reiterate too much, but I think that you are correct here:
NotImplemented is not Pythonic. It's a complete hack that I would think
people would like to quarantine to the limited use case of binary
operations. Even with the poorly contextualized email that Guido received,
I think it was clear that he thought it wasn't appropriate outside of that
use case either (I don't think better context would have changed that).

Personally, I was willing to overlook that for the sake of a speedy
resolution. But reading the email chain, it seems that only Thomas and
Daniel are currently in favor of using NotImplemented, which is a shift
from the earlier situation. So now it seems that NotImplemented is not
likely to be used.

2017-08-29 14:07 GMT-05:00 Paul Moore :

> On 29 August 2017 at 19:45, Chris Barker  wrote:
> > But it would be nice if packages didn't NEED to be pip installed -- but I
> > think that's an issue with the wheel spec (that may well be solved).
>
> Wheels don't need pip to install them - the spec has always been clear
> on how to "manually" install a wheel (unzip it and move a few things
> around), and the wheel package itself has a simple "wheel install"
> command.
>
> If you want to create wrapper scripts for entry points, you need to do
> a bit more work, but distlib offers that functionality (that's what
> pip uses).
>
> One point about PEP 517 is that it provides a standard way to ask
> *any* build system to make a wheel. So it's in theory possible to
> install any package without pip (of course, if you try to handle all
> the various corner cases, extra quirks, etc that come up in the real
> world, you end up reinventing pip ;-)).
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 29 August 2017 at 19:45, Chris Barker  wrote:
> But it would be nice if packages didn't NEED to be pip installed -- but I
> think that's an issue with the wheel spec (that may well be solved).

Wheels don't need pip to install them - the spec has always been clear
on how to "manually" install a wheel (unzip it and move a few things
around), and the wheel package itself has a simple "wheel install"
command.

If you want to create wrapper scripts for entry points, you need to do
a bit more work, but distlib offers that functionality (that's what
pip uses).

One point about PEP 517 is that it provides a standard way to ask
*any* build system to make a wheel. So it's in theory possible to
install any package without pip (of course, if you try to handle all
the various corner cases, extra quirks, etc that come up in the real
world, you end up reinventing pip ;-)).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

> conda-build doesn't produce or consume wheels.  It works by creating a
> clean
> > conda environment and running a shell script to install the python
> package
> > into that environment


To be really clear -- conda build doesn't directly use ANY build/install
system.

All files produced by the install
> > script are then put into a tar file, which is conda's version of a wheel.
>

right -- conda-build simply runs the script it is told to run (by the
recipe), and then looks and sees what files were installed.

In theory, one could build a conda package directly out of a binary wheel
-- after all, as mentioned, they are actually pretty similar, but most
folks would probably just use pip inside their conda-build recipe.

In fact, that is already being done -- most python packages are built with
"setup.py install", but some are built by using pip to install a wheel.

But I didn't think that the goal of PEP 517 was to ensconce pip as the only
way to install a package in Python.

So if a build system can install (like setuptools currently does), then
conda recipes may well call it directly.

I don't think this really has any bearing on the the PEP, though -- As I
understand it, the goal of the PEP is to formalize the interface between
build systems and package managers -- and that can only be a good thing for
other package managers.


NOTE: For the most part, folks want to use a pip-compatible install in
conda-build, because users tend to also directly use pip to install stuff
in conda environments e.g. if there is not a conda package available. If a
conda package were built with, say, plain distutils, then pip won't find
the meta data it's looking for, and might falsely see an unresolved
dependency.

Personally, I've always avoided pip installing packages in conda --
preferring to make conda packages for all the deps I need, but conda-pip
compatibility has come a long way, so it does work fine for many users.

But it would be nice if packages didn't NEED to be pip installed -- but I
think that's an issue with the wheel spec (that may well be solved).

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 29 August 2017 at 17:22, Phil Austin  wrote:
> On 2017-08-29 01:20, Paul Moore wrote:
>>
>> On 29 August 2017 at 01:13, Chris Barker  wrote:
>>> If the build API is clean and supported enough, I can see conda using it.
>>>
>>> just sayin'
>>
>>
>> Thanks - that's really good to know. So *is* the build API clean
>> enough for you? Specifically:
>>
>> 1. build_sdist can be missing, or can report back (somehow) that it
>> failed to produce a sdist, and you should fall back to build_wheel.
>> 2. build_wheel must always succeed and must always exist.
>> 3. All hooks must be run in a separate subprocess
>>
>> There's also error handling, which I don't recall the details of, but
>> I think boils down to the backend code can write what it likes to
>> stdout/stderr and may raise an exception indicating "something blew
>> up".
>>
>> Those are basically the key points at the moment. Add to that, does
>> conda mind if build_wheel might result in a different wheel than
>> build_sdist->build_wheel would produce?
>
> conda-build doesn't produce or consume wheels.  It works by creating a clean
> conda environment and running a shell script to install the python package
> into that environment (including its own versions of /usr/lib /usr/etc
> /usr/bin  /urs/include /sbin -- which is why it can cope with non-python
> packages like g++, julia, R, git etc.).   All files produced by the install
> script are then put into a tar file, which is conda's version of a wheel.

OK, so I don't see how conda would use PEP 517 the way Chris
suggested. But never mind - if it doesn't help conda that's fine.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Phil Austin]

On 2017-08-29 01:20, Paul Moore wrote:

On 29 August 2017 at 01:13, Chris Barker  wrote:

On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore  wrote:


Me too. At the moment, I only expect two backends of any substance -
your flit backend and xoviat's setuptools one. But I only know of one
frontend, namely pip - and talk of projects like tox or twine acting
as frontends never seems to get any traction.


scikit-build also expressed some interest:

https://www.mail-archive.com/distutils-sig@python.org/msg26553.html

although I don't see the pull request.






If the build API is clean and supported enough, I can see conda using it.

just sayin'


Thanks - that's really good to know. So *is* the build API clean
enough for you? Specifically:

1. build_sdist can be missing, or can report back (somehow) that it
failed to produce a sdist, and you should fall back to build_wheel.
2. build_wheel must always succeed and must always exist.
3. All hooks must be run in a separate subprocess

There's also error handling, which I don't recall the details of, but
I think boils down to the backend code can write what it likes to
stdout/stderr and may raise an exception indicating "something blew
up".

Those are basically the key points at the moment. Add to that, does
conda mind if build_wheel might result in a different wheel than
build_sdist->build_wheel would produce?



conda-build doesn't produce or consume wheels.  It works by creating a 
clean conda environment and running a shell script to install the python 
package into that environment (including its own versions of /usr/lib 
/usr/etc /usr/bin  /urs/include /sbin -- which is why it can cope with 
non-python packages like g++, julia, R, git etc.).   All files produced 
by the install script are then put into a tar file, which is conda's 
version of a wheel.



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Brett Cannon]

On Mon, 28 Aug 2017 at 16:29 Nathaniel Smith  wrote:

> On Mon, Aug 28, 2017 at 1:27 PM, Thomas Kluyver 
> wrote:
> > On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote:
> >
> > > Then end the debate by letting the PEP authors decide the return type,
> and
> > > write a paragraph explaining why the other options were rejected. It
> is not
> > > going to make a big difference.
> >
> >
> > Will that work now? Are we all so tired of this endless war that people
> will
> > sign a peace treaty written by the people whose names are on the PEP
> > (Nathaniel & me)?
> >
> > If so, let the trumpets sound, and the heralds declare that "return
> > NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's
> > preference right ;-)
>
> Fine with me, though if it turns out Donald and Nick prefer the
> version where the backend has to export an exception class then I'm
> fine with that too. (I'm basing this on -- Donald has said he likes
> it, and Nick hasn't commented yet but AFAICT it does address his
> concerns with NotImplemented, so it seem like a plausible outcome.)
>

I loathe to weigh in on this and add yet another voice in this discussion,
but the exported exception seems like the best solution for everyone
involved from my lurking perspective. For me, using NotImplemented is a
misuse of the singleton since I know what it's meant to be used for (and so
I cringe every time I hear it brought up as a solution). And I was fine
with NotImplementedError but if people want something more specific and
None is out due to worries of accidental bare returns, then the exported
exception comes the closest to making everyone happy (it does tick the
EIBTI box :) .
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 29 August 2017 at 01:13, Chris Barker  wrote:
> On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore  wrote:
>>
>> Me too. At the moment, I only expect two backends of any substance -
>> your flit backend and xoviat's setuptools one. But I only know of one
>> frontend, namely pip - and talk of projects like tox or twine acting
>> as frontends never seems to get any traction.
>
>
> If the build API is clean and supported enough, I can see conda using it.
>
> just sayin'

Thanks - that's really good to know. So *is* the build API clean
enough for you? Specifically:

1. build_sdist can be missing, or can report back (somehow) that it
failed to produce a sdist, and you should fall back to build_wheel.
2. build_wheel must always succeed and must always exist.
3. All hooks must be run in a separate subprocess

There's also error handling, which I don't recall the details of, but
I think boils down to the backend code can write what it likes to
stdout/stderr and may raise an exception indicating "something blew
up".

Those are basically the key points at the moment. Add to that, does
conda mind if build_wheel might result in a different wheel than
build_sdist->build_wheel would produce?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Mon, Aug 28, 2017 at 12:50 PM, Paul Moore  wrote:

> Me too. At the moment, I only expect two backends of any substance -
> your flit backend and xoviat's setuptools one. But I only know of one
> frontend, namely pip - and talk of projects like tox or twine acting
> as frontends never seems to get any traction.
>

If the build API is clean and supported enough, I can see conda using it.

just sayin'

NOTE: it wouldn't be conda per-se using it -- but individual conda recipes,
which generally call setuptools directly now.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Mon, Aug 28, 2017 at 12:20 PM, Paul Moore  wrote:

> > interface). So if pip calls build_sdist and then build_wheel, there will
> be
> > two source distributions built (one by pip and one by setuptools) before
> > building a wheel.


exactly why pip should NOT concern itself with the sdist -- either the back
end needs to do it anyway, or it doesn't need to do it at all, and would
only be doing so to satisfy pip.


> This is precisely the question - whose responsibility is it to ensure
> that sdist->wheel vs build_wheel equivalence is maintained? The PEP
> says it's the backend's responsibility, so pip shouldn't need to do
> anything.
>

makes sense to me.

But, we're currently having debates about "return None" which hinge on
> "if backends have bugs, return None might not work well". Sure - but
> if backends have bugs, they may not maintain wheel equivalence.


If backends have bugs (and they will), any number of things can go wrong.
Making the spec so that front ends can have a better idea what went wrong
is a fine idea, but it can only go so far.

The only thing we really need to distinguish is " that functionality is not
implemented" from "something went wrong"

And not even that if we don't have pip trying to go through some
machinations about what do if the back-end does not produce a sdist.


>  To that extent, I'm with Donald - pip going
> sdist->wheel protects the user against the known-to-be-an-issue bug
> that the backend doesn't ensure wheel equivalence.


pip can not protect the user from a poorly written back-end. And it
shouldn't try.


* Chris Barker has pointed out that backends have no reason to support
> sdists now.
>

not quite -- the reason to support sdist is because you want an sdist, not
because it's a necessary part of the path to a wheel. I don't quite
understand your (Paul's) point about the importance of sdists to open
source (as opposed to a plain old tarball of the source, like what gitHub
produces when you do a release.

* Nathaniel is pushing a means of notifying "I can't build a sdist"
> that protects against backends accidentally not following the spec.
>

"I can't do that" respond to any possible request makes sense. Unless "I
can't build a sdist" violates the protocol -- which I don't think it should.

> Should we trust the backend or not? Backends *will* have bugs - part

> of "trust the backend" is simply telling the user that the problem
> behaviour they found is not pip's issue and should be reported to the
> backend. Is that a sufficiently bad user experience that we should try
> to improve it (experience with setuptools says that it is - why are we
> assuming that developers of new backends will be so much more
> conscientious and careful than the setuptools developers? *I*
> certainly don't feel that the setuptools developers are unusually bad
> - quite the opposite!)
>

If that is a sufficiently bad user experience then the only other option is
for "us" by some definition of "us" to build the back-end, too -- which is
where this all started with setuptool. But my understanding is that goal is
was not to force the whole stack to maintained together.

And really, the end-user will need to report the problem to the package
maintainer, not the build tool maintainer jsu tlike they should no if
something doesn't pip install.

Having a more robust means of saying "I can't build a sdist" than
> "return None" is protecting the user from issues with the backend. So
> is building via sdist.
>

but only one particular set of issues -- when there are an infinite number
of possible issues...

One other thought --

the easiest way to make an sdist it to simply copy the source tree.

So:

1) not a big deal to require back-ends to do it -- it's easy to do
but
2) back ends may do it the easy and sloppy way, and get build artifacts in
the sdist, and then you are back where you started.

so requiring the sdist step in no way saves you from poorly written
back-ends. It may make it worse, as you _think_ you are getting something
clean.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Mon, Aug 28, 2017 at 1:27 PM, Thomas Kluyver  wrote:
> On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote:
>
> > Then end the debate by letting the PEP authors decide the return type, and
> > write a paragraph explaining why the other options were rejected. It is not
> > going to make a big difference.
>
>
> Will that work now? Are we all so tired of this endless war that people will
> sign a peace treaty written by the people whose names are on the PEP
> (Nathaniel & me)?
>
> If so, let the trumpets sound, and the heralds declare that "return
> NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's
> preference right ;-)

Fine with me, though if it turns out Donald and Nick prefer the
version where the backend has to export an exception class then I'm
fine with that too. (I'm basing this on -- Donald has said he likes
it, and Nick hasn't commented yet but AFAICT it does address his
concerns with NotImplemented, so it seem like a plausible outcome.)

I hope Nick had a good weekend at the beach or whatever, because this
is going to be a heck of an email backlog to come back to...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> If so, let the trumpets sound, and the heralds declare that "return
NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's
preference right ;-)

For better or for worse, the trumpets seem to be sounding against this idea
(Nathaniel seemed okay with whatever Donald and Nick thought was
appropriate).

2017-08-28 15:27 GMT-05:00 Thomas Kluyver :

> On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote:
>
> Then end the debate by letting the PEP authors decide the return type, and
> write a paragraph explaining why the other options were rejected. It is not
> going to make a big difference.
>
>
> Will that work now? Are we all so tired of this endless war that people
> will sign a peace treaty written by the people whose names are on the PEP
> (Nathaniel & me)?
>
> If so, let the trumpets sound, and the heralds declare that "return
> NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's
> preference right ;-)
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Thomas Kluyver]

On Mon, Aug 28, 2017, at 09:13 PM, Daniel Holth wrote:
> Then end the debate by letting the PEP authors decide the return type,
> and write a paragraph explaining why the other options were rejected.
> It is not going to make a big difference.
Will that work now? Are we all so tired of this endless war that people
will sign a peace treaty written by the people whose names are on the
PEP (Nathaniel & me)?
If so, let the trumpets sound, and the heralds declare that "return
NotImplemented" is the way to do it. (I hope I've remembered Nathaniel's
preference right ;-)
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Aug 28, 2017, at 3:59 PM, Thomas Kluyver  wrote:
> 
> The difference I see with the "return None" question is that there we
> have an alternative (return NotImplemented) which is just as simple for
> both sides, but avoids the identified issue with a buggy backend. The
> only argument there seems to be for using None is about semantics - and
> that's not a great argument, because 'practicality beats purity'.
> 
> With the questions over sdist/wheel consistency, there's a complexity
> cost, for the spec and for frontends, in deciding that they can't. So
> we're weighing a trade-off: do we force ourselves to resolve the
> notimplemented question so that frontends can do sdist-wheel+fallback,
> or do we leave it up to frontends and risk some bugs which we might
> otherwise have prevented.


I think the best option is to just export an exception that a frontend can 
catch. That gives like 95% of the benefit of either mechanism for an extra 5% 
of work. If a backend is unwilling to add a:

class UnsupportedOperation(Exception):
pass

To their backend, then they’re likely not a backend that we should be spending 
a whole lot of time or effort trying to support. Two copy/pasteable lines of 
code is a tiny price to pay. I haven’t talked to Nick, but I’d be surprised if 
he was against this, since it’s basically what he wanted with a more specific 
exception (which is good practice anyways).

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Aug 28, 2017, at 3:58 PM, Paul Moore  wrote:
> 
> On 28 August 2017 at 20:47, Donald Stufft  wrote:
>> I also believe it is fundamentally impossible for the backends to guarantee
>> consistency if they have a separate list for what gets installed vs what
>> gets put into a sdist without literally building a sdist (or something
>> similar)— and as I understand it one of the issues with that is that the
>> tools don’t want to mandate being able to do that same logic in all
>> situations.
> 
> And yet that's precisely what xoviat is intending to do, based on what
> the PEP says is needed. Are you saying he shouldn't? And the
> setuptools should simply expose the same consistency problems we've
> been dealing with until now? And the solution is for pip to switch to
> sdist->wheel build as we've always planned in order to resolve this
> issue? And any other future frontends will have to go through the same
> cycle that pip did?


I had some non-OSS stuff come up so I lost track of the discussion part way 
through so maybe I missed this recommendation, but I think that is going to be 
a bad path to go down because it is intermixing things in a way that I feel is 
going to result in a less optimal outcome. The problem is the backend doesn’t 
really know why we’re asking it to build a wheel to know whether we care or not 
about consistency guarantees. 

Future front ends can make their own decisions as it makes sense for their 
workflow. Something like Debian might choose to skip the sdist option (when 
they go straight from a VCS archive or w/e) because they have a human being 
sitting there ensuring that the install looks correct, and they are unlikely to 
want to go through sdist. 

> 
> In which case, isn't the issue here that we've failed to argue
> sufficiently persuasively that all backends must provide a
> non-optional build_sdist hook, so we can do that cleanly?


I’m not sure what you mean by non-optional— Do you mean a build_Sdist hook that 
does not have a “not implemented” flag? That would probably be my preference, 
but I’m happy to compromise on the not implemented return 
value/exception/whatever as a middle ground. Honestly this PEP has been drawn 
out enough with enough hypotheticals and corner cases that one of the main 
things I’m trying to optimize for is flexibility so we can handle different 
possible scenarios without going through this again.


> 
>> So I don’t think it works to say “You *must* ensure a consistent
>> output, and I think the only thing we can do is say that you *SHOULD* try to
>> be consistent, and leave it up to front ends to decide how seriously they
>> take that as a requirement.
> 
> But how do frontends decide that? They have to base the decision on
> experience with backends. And it looks like the current crop of
> backends are split down the middle. Remember that if pip decides to go
> sdist->wheel, then that strongly motivates xoviat to abandon his
> current approach to ensure consistency, as it's duplicating the work
> pip will do. At which point the setuptools backend can't be assumed to
> be consistent, and the flit backend must be assumed to be (because we
> can't guarantee it can make a sdist).
> 
> 


Well I don’t think it’s solely based on their experience with backends, but 
also what the general goal of their tool is. Something that is designed to run 
in CI might take a very strict stance to be extra sure to catch errors, 
something that runs in cases where you care more about performance than 
correctness in edge cases or specialized tools where they’ve done extra work 
and aren’t processing arbitrary Packages might just skip straight to building 
wheels.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

Then end the debate by letting the PEP authors decide the return type, and
write a paragraph explaining why the other options were rejected. It is not
going to make a big difference.

On Mon, Aug 28, 2017 at 3:59 PM Thomas Kluyver  wrote:

> On Mon, Aug 28, 2017, at 08:50 PM, Paul Moore wrote:
> > My main motivation for wavering is that I thought agreeing to trust
> > the backend would simplify many of the decisions, and it's immensely
> > frustrating to me that we're still debating the same question in the
> > "return None" thread.
>
> The difference I see with the "return None" question is that there we
> have an alternative (return NotImplemented) which is just as simple for
> both sides, but avoids the identified issue with a buggy backend. The
> only argument there seems to be for using None is about semantics - and
> that's not a great argument, because 'practicality beats purity'.
>
> With the questions over sdist/wheel consistency, there's a complexity
> cost, for the spec and for frontends, in deciding that they can't. So
> we're weighing a trade-off: do we force ourselves to resolve the
> notimplemented question so that frontends can do sdist-wheel+fallback,
> or do we leave it up to frontends and risk some bugs which we might
> otherwise have prevented.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Thomas Kluyver]

On Mon, Aug 28, 2017, at 08:50 PM, Paul Moore wrote:
> My main motivation for wavering is that I thought agreeing to trust
> the backend would simplify many of the decisions, and it's immensely
> frustrating to me that we're still debating the same question in the
> "return None" thread.

The difference I see with the "return None" question is that there we
have an alternative (return NotImplemented) which is just as simple for
both sides, but avoids the identified issue with a buggy backend. The
only argument there seems to be for using None is about semantics - and
that's not a great argument, because 'practicality beats purity'.

With the questions over sdist/wheel consistency, there's a complexity
cost, for the spec and for frontends, in deciding that they can't. So
we're weighing a trade-off: do we force ourselves to resolve the
notimplemented question so that frontends can do sdist-wheel+fallback,
or do we leave it up to frontends and risk some bugs which we might
otherwise have prevented.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 28 August 2017 at 20:47, Donald Stufft  wrote:
> I also believe it is fundamentally impossible for the backends to guarantee
> consistency if they have a separate list for what gets installed vs what
> gets put into a sdist without literally building a sdist (or something
> similar)— and as I understand it one of the issues with that is that the
> tools don’t want to mandate being able to do that same logic in all
> situations.

And yet that's precisely what xoviat is intending to do, based on what
the PEP says is needed. Are you saying he shouldn't? And the
setuptools should simply expose the same consistency problems we've
been dealing with until now? And the solution is for pip to switch to
sdist->wheel build as we've always planned in order to resolve this
issue? And any other future frontends will have to go through the same
cycle that pip did?

In which case, isn't the issue here that we've failed to argue
sufficiently persuasively that all backends must provide a
non-optional build_sdist hook, so we can do that cleanly?

> So I don’t think it works to say “You *must* ensure a consistent
> output, and I think the only thing we can do is say that you *SHOULD* try to
> be consistent, and leave it up to front ends to decide how seriously they
> take that as a requirement.

But how do frontends decide that? They have to base the decision on
experience with backends. And it looks like the current crop of
backends are split down the middle. Remember that if pip decides to go
sdist->wheel, then that strongly motivates xoviat to abandon his
current approach to ensure consistency, as it's duplicating the work
pip will do. At which point the setuptools backend can't be assumed to
be consistent, and the flit backend must be assumed to be (because we
can't guarantee it can make a sdist).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

On Mon, Aug 28, 2017 at 3:48 PM xoviat  wrote:

> > But I'm suspicious of the rationale that *there will be fewer frontends so
> they should have more responsibility*.
>
> To be fair, pip is currently struggling to keep up with project
> requirements as it is, which has caused some frustration in the community
> (not that the frustration isn't wrong, but I don't think it's not going to
> resolve any problems). I don't see how having even more frontends will
> resolve these manpower issues.
>

Simple. It would be different people.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 28 August 2017 at 20:32, Thomas Kluyver  wrote:
> On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
>> Maybe we go fully to Nick's proposal that we don't mandate any sort of
>> consistency constraints in the PEP. That would mean pip *has* to go
>> sdist->wheel (because pip does need consistent behaviour), and
>> xoviat's setuptools backend can skip building a sdist on the way to
>> building a wheel. It also means that the build_wheel hook is
>> essentially unreliable in isolation, and that all frontends will
>> likely have to do the build_sdist->build_wheel with fallback to
>> inplace build_wheel dance that pip does. But we expect many less
>> frontends than backends, so maybe that's the right trade-off?
>
> I've mentioned this before, but I have little faith in our ability to
> predict that one side of an interface will be far more numerous than the
> other. This is partly from my experience with Jupyter, where we got that
> prediction completely wrong. But I also think it's quite plausible that
> many frontend tools will want to use this interface (to argue against
> myself, most will be Python tools, so they could theoretically share a
> common wrapper module - but I have some doubts about whether they will).
>
> I'm not saying we shouldn't do this - as Donald is firmly in favour and
> you're wavering, it seems the easier option to wrap the discussion up.
> But I'm suspicious of the rationale that *there will be fewer frontends
> so they should have more responsibility*.

Me too. At the moment, I only expect two backends of any substance -
your flit backend and xoviat's setuptools one. But I only know of one
frontend, namely pip - and talk of projects like tox or twine acting
as frontends never seems to get any traction.

The problem is that xoviat has to do a lot of work to provide
guarantees that flit gets essentially for free, but flit can't provide
the guaranteed sdist->wheel route that would let pip just go
sdist->wheel cleanly. So there's a mismatch of requirements and
*someone* needs to plaster over the cracks. At the moment, the debate
is essentially whether that's xoviat or pip. Donald wants it to be pip
because we can't guarantee that the next backend developer will be
aware of all of the issues we debated here, and so we have to protect
against that. I was persuaded[1] that we could document the
requirements in the spec and expect that to be enough.

I do worry that there's a risk that we're crippling the PEP with
sufficient complexity that no-one will bother writing alternative
frontends *or* additional backends (although I hope your support
library might ease that problem somewhat).

My main motivation for wavering is that I thought agreeing to trust
the backend would simplify many of the decisions, and it's immensely
frustrating to me that we're still debating the same question in the
"return None" thread.

Paul

[1] I don't say "I thought" because I *never* thought backends would
be sufficiently reliable. I just accepted that my concerns are only
hypothetical. I wish others could do the same :-(
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

> But I'm suspicious of the rationale that *there will be fewer frontends so
they should have more responsibility*.

To be fair, pip is currently struggling to keep up with project
requirements as it is, which has caused some frustration in the community
(not that the frustration isn't wrong, but I don't think it's not going to
resolve any problems). I don't see how having even more frontends will
resolve these manpower issues.

2017-08-28 14:32 GMT-05:00 Thomas Kluyver :

> On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
> > Maybe we go fully to Nick's proposal that we don't mandate any sort of
> > consistency constraints in the PEP. That would mean pip *has* to go
> > sdist->wheel (because pip does need consistent behaviour), and
> > xoviat's setuptools backend can skip building a sdist on the way to
> > building a wheel. It also means that the build_wheel hook is
> > essentially unreliable in isolation, and that all frontends will
> > likely have to do the build_sdist->build_wheel with fallback to
> > inplace build_wheel dance that pip does. But we expect many less
> > frontends than backends, so maybe that's the right trade-off?
>
> I've mentioned this before, but I have little faith in our ability to
> predict that one side of an interface will be far more numerous than the
> other. This is partly from my experience with Jupyter, where we got that
> prediction completely wrong. But I also think it's quite plausible that
> many frontend tools will want to use this interface (to argue against
> myself, most will be Python tools, so they could theoretically share a
> common wrapper module - but I have some doubts about whether they will).
>
> I'm not saying we shouldn't do this - as Donald is firmly in favour and
> you're wavering, it seems the easier option to wrap the discussion up.
> But I'm suspicious of the rationale that *there will be fewer frontends
> so they should have more responsibility*.
>
> Thomas
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Aug 28, 2017, at 3:32 PM, Thomas Kluyver  wrote:
> 
> On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
>> Maybe we go fully to Nick's proposal that we don't mandate any sort of
>> consistency constraints in the PEP. That would mean pip *has* to go
>> sdist->wheel (because pip does need consistent behaviour), and
>> xoviat's setuptools backend can skip building a sdist on the way to
>> building a wheel. It also means that the build_wheel hook is
>> essentially unreliable in isolation, and that all frontends will
>> likely have to do the build_sdist->build_wheel with fallback to
>> inplace build_wheel dance that pip does. But we expect many less
>> frontends than backends, so maybe that's the right trade-off?
> 
> I've mentioned this before, but I have little faith in our ability to
> predict that one side of an interface will be far more numerous than the
> other. This is partly from my experience with Jupyter, where we got that
> prediction completely wrong. But I also think it's quite plausible that
> many frontend tools will want to use this interface (to argue against
> myself, most will be Python tools, so they could theoretically share a
> common wrapper module - but I have some doubts about whether they will).
> 
> I'm not saying we shouldn't do this - as Donald is firmly in favour and
> you're wavering, it seems the easier option to wrap the discussion up.
> But I'm suspicious of the rationale that *there will be fewer frontends
> so they should have more responsibility*.
> 


I don’t see anything wrong with saying *both* sides should be doing this when 
they can. Backends should attempt to be consistent where possible— while 
recognizing that is not 100% possible in every situation (for instance, flit 
when you’re in an archive from github.com  w/o .git).

Likewise front ends should decide how important that is to them, which can 
range from “not important at all, YOLO” that just blindly calls build_wheel, to 
“We’d like it, but we aren’t going to mandate it” that calls build_sdist and 
falls back to build_wheel, and “We think this is a hard requirement” that calls 
build_sdist and fails if it can’t.

My proposal isn’t exactly that I don’t trust the backends, it is that we don’t 
really know what the landscape is going to look at. Thus this proposal allows 
us to implement all three possible options without changing anything without 
mandating something that isn’t feasible (all backends must always be 100% 
consistent all the time).

It would then be my desire that pip starts off with a pretty safe middle 
ground, try to build a sdist and fall back to directly building wheels that (A) 
helps to prevent issues where backends may have bugs or inadvertently are 
violating the ideal of a consistent build and (B) succeeds in most cases, even 
if (A) can’t be satisfied. Since we have flexibility we can then watch what 
happens, and if we find that a lot of problems are cropping up from 
inconsistent builds, then we can start talking about moving to mandating going 
via sdist OR if we find that the crop of build tools do a really good job and 
we think that we don’t really need the sanity check, migrate to just going 
straight to wheel.

I think this is pretty important too, because how you want to handle not being 
able to create a sdist is going to be fairly task specific. Obviously if you 
can’t produce a sdist for a hypothetical ``twine sdist`` or ``pip sdist`` 
command, that is going to be a hard failure, but for ``pip install`` or ``pip 
wheel``, then there is a pretty reasonable argument that a fallback or straight 
to wheel is the correct answer.

I also believe it is fundamentally impossible for the backends to guarantee 
consistency if they have a separate list for what gets installed vs what gets 
put into a sdist without literally building a sdist (or something similar)— and 
as I understand it one of the issues with that is that the tools don’t want to 
mandate being able to do that same logic in all situations. So I don’t think it 
works to say “You *must* ensure a consistent output, and I think the only thing 
we can do is say that you *SHOULD* try to be consistent, and leave it up to 
front ends to decide how seriously they take that as a requirement.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Thomas Kluyver]

On Mon, Aug 28, 2017, at 08:20 PM, Paul Moore wrote:
> Maybe we go fully to Nick's proposal that we don't mandate any sort of
> consistency constraints in the PEP. That would mean pip *has* to go
> sdist->wheel (because pip does need consistent behaviour), and
> xoviat's setuptools backend can skip building a sdist on the way to
> building a wheel. It also means that the build_wheel hook is
> essentially unreliable in isolation, and that all frontends will
> likely have to do the build_sdist->build_wheel with fallback to
> inplace build_wheel dance that pip does. But we expect many less
> frontends than backends, so maybe that's the right trade-off?

I've mentioned this before, but I have little faith in our ability to
predict that one side of an interface will be far more numerous than the
other. This is partly from my experience with Jupyter, where we got that
prediction completely wrong. But I also think it's quite plausible that
many frontend tools will want to use this interface (to argue against
myself, most will be Python tools, so they could theoretically share a
common wrapper module - but I have some doubts about whether they will).

I'm not saying we shouldn't do this - as Donald is firmly in favour and
you're wavering, it seems the easier option to wrap the discussion up.
But I'm suspicious of the rationale that *there will be fewer frontends
so they should have more responsibility*.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 28 August 2017 at 19:50, xoviat  wrote:
> Personally, my plan for the setuptools backend will be to build a source
> distribution (essentially using the command-line interface), extract into a
> tmpdir, and then build a wheel (essentially using the command line
> interface). So if pip calls build_sdist and then build_wheel, there will be
> two source distributions built (one by pip and one by setuptools) before
> building a wheel. There is not another way to do this that will comply with
> the specification because setuptools cannot currently be trusted to build a
> wheel directly.

This is precisely the question - whose responsibility is it to ensure
that sdist->wheel vs build_wheel equivalence is maintained? The PEP
says it's the backend's responsibility, so pip shouldn't need to do
anything.

But, we're currently having debates about "return None" which hinge on
"if backends have bugs, return None might not work well". Sure - but
if backends have bugs, they may not maintain wheel equivalence. I find
this discrepancy immensely frustrating, as in one breath we're being
told "trust the backend", and in the next we're being told "the
backend may have bugs". To that extent, I'm with Donald - pip going
sdist->wheel protects the user against the known-to-be-an-issue bug
that the backend doesn't ensure wheel equivalence. But if we go with
that, conscientious backend developers like xoviat suffer, because
they take extra steps to do things correctly and as a result suffer
bad performance.

I've said I'm happy to trust the backend. But I'm starting to wonder
if I should change that position:

* Chris Barker has pointed out that backends have no reason to support
sdists now.
* Nathaniel is pushing a means of notifying "I can't build a sdist"
that protects against backends accidentally not following the spec.
* Donald has reservations (I don't fully understand them, except in
the broad sense of "he wants pip to protect the user from problem
cases" - which basically means "he doesn't trust the backend" - but in
general I think they are valid)

Should we trust the backend or not? Backends *will* have bugs - part
of "trust the backend" is simply telling the user that the problem
behaviour they found is not pip's issue and should be reported to the
backend. Is that a sufficiently bad user experience that we should try
to improve it (experience with setuptools says that it is - why are we
assuming that developers of new backends will be so much more
conscientious and careful than the setuptools developers? *I*
certainly don't feel that the setuptools developers are unusually bad
- quite the opposite!)

Having a more robust means of saying "I can't build a sdist" than
"return None" is protecting the user from issues with the backend. So
is building via sdist.

Donald's position is consistent - build via sdist and use a robust
return. So was mine - build_wheel and return None. I don't want the
PEP to end up defining an inconsistent position. And I'm starting to
feel that conceding on build_wheel (and the "trust the backend"
principle) was not seen in that context, but as conceding just the one
item, leaving other aspects of trusting the backend open to debate.

Maybe we go fully to Nick's proposal that we don't mandate any sort of
consistency constraints in the PEP. That would mean pip *has* to go
sdist->wheel (because pip does need consistent behaviour), and
xoviat's setuptools backend can skip building a sdist on the way to
building a wheel. It also means that the build_wheel hook is
essentially unreliable in isolation, and that all frontends will
likely have to do the build_sdist->build_wheel with fallback to
inplace build_wheel dance that pip does. But we expect many less
frontends than backends, so maybe that's the right trade-off?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 28 August 2017 at 17:25, Chris Barker  wrote:
> Nor should it be for any other system. Before I saw this post I was about to
> make a point about not contorting ourselves to make things works the same
> way as pip and setuptools currently work -- pip was kind of hacked together
> to support setuptools, and setuptools grew kinda-organically from distutils
> (while also shoehorning in other not-really-building functionality).
>
> So let's make the "new" API clean and adapt the current tools to that.
>
> Specifically, I'm really confused about the sdist concept -- sure, it's a
> nice idea to have a standard way to create a source distributions, and I can
> see that tools that, for instance, create uploads for PyPi and the like will
> want to do that, but I don't see it as an inherent part of building a binary
> distribution (i.e. wheel), or building and installing a package directly.

This is *precisely* why I remain uncomfortable about the "just call
build_wheel" approach. I agree (as I said) that going sdist->wheel in
theory should make no difference, assuming that the backend follows
the PEP. And so I'm in that sense persuaded that pip doesn't need to
go sdist->wheel.

However, I am *strongly* against allowing backends to skip the concept
of a sdist altogether. Sufficiently so that I refused to use or
support flit prior to it gaining support for building sdists. There's
a problem of timing at the moment, in that we have not yet defined a
new sdist format, so a sdist is currently loosely defined based on
"what distutils makes, with some adhoc modifications to incorporate
pyproject.toml. That's deliberate - we wanted to avoid PEP 517 getting
bogged down in "sdist 2.0" discussions. But none of that matters to me
- Python is an open source project, and a packaging infrastructure
that isn't built around publishing sources is not something I want to
support. And part of that is that we should be expecting build systems
to support source publication, and expect projects to publish sources
on PyPI (I do *not* consider a link to a project's github site, or
other source repository, as sufficient - people can close their github
accounts, projects can be abandoned - but PyPI should *always* contain
what's needed to reproduce the distributions on there - and that means
sources).

So while I concede that pip might not *need* to build wheels via
sdist, that does *not* mean I am in any way in favour of omitting
"build a sdist" from the PEP 517 spec. I'm uncomfortable enough with
allowing backends to have the ability to build a wheel when they can't
build a sdist (and yes, that means I have reservations about how flit
supports sdists). But for me, having a hook to build a sdist is
non-negotiable. We've had debates in the past about whether building a
sdist is in scope for the PEP - I do *not* want the discussion to go
back to the point where we open up that question again.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Personally, my plan for the setuptools backend will be to build a source
distribution (essentially using the command-line interface), extract into a
tmpdir, and then build a wheel (essentially using the command line
interface). So if pip calls build_sdist and then build_wheel, there will be
two source distributions built (one by pip and one by setuptools) before
building a wheel. There is not another way to do this that will comply with
the specification because setuptools cannot currently be trusted to build a
wheel directly.

2017-08-28 12:15 GMT-05:00 Donald Stufft :

>
> On Aug 28, 2017, at 12:29 PM, Chris Barker  wrote:
>
> On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft  wrote:
>
>> Donald, what do you think? IIRC, you were most keen on going
>> sdist->wheel where possible, and I don't think you've commented on
>> Paul's suggestion yet (apologies if I've overlooked a response).
>>
>>
>> I still think it should, and prefer pip to attempt to build a sdist prior
>> to building a wheel when we’re coming from a VCS directory. I think that is
>> going to be the most robust mechanism with the least amount of surprising
>> behavior for end users.
>>
>
> why? even as it stands, doesn't setuptools dump everything into the build
> dir anyway? The "Creating of an sdist" really seems like a build-system
> problem, not a package manager problem to me.
>
> > "when we’re coming from a VCS directory."
>
> whether this is a VCS directory or some other source seems to me like
> something pip should not need to know...
>
>
>
> Differences between what files are in a sdist and what files are in a VCS
> directory will lead to different behaviors on install which makes ``pip
> install .`` and ``build-a-sdist && pip install the-sdist.tar.gz`` behave
> differently. Attempting to funnel everything through the same VCS -> sdist
> -> wheel path makes it less likely for these kinds of issues to occur.
>
> This is not a fully resolvable problem, and it is going to happen
> basically anytime you have two independent lists of what files get put into
> a sdist and what files get installed. It is not unique to setuptools nor is
> it a result of the way distutils/setuptools works— they only expose it more
> obviously because of their relevant APIs. Infact, both enscons and flit
> have this same problem (although flit has gone to some length to minimize
> the issue, so it’s somewhat hard, but not impossible, to actually trigger
> it).
>
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Aug 28, 2017, at 12:29 PM, Chris Barker  wrote:
> 
> On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft  > wrote:
>> Donald, what do you think? IIRC, you were most keen on going
>> sdist->wheel where possible, and I don't think you've commented on
>> Paul's suggestion yet (apologies if I've overlooked a response).
> 
> 
> I still think it should, and prefer pip to attempt to build a sdist prior to 
> building a wheel when we’re coming from a VCS directory. I think that is 
> going to be the most robust mechanism with the least amount of surprising 
> behavior for end users.
> 
> why? even as it stands, doesn't setuptools dump everything into the build dir 
> anyway? The "Creating of an sdist" really seems like a build-system problem, 
> not a package manager problem to me.
> 
> > "when we’re coming from a VCS directory."
> 
> whether this is a VCS directory or some other source seems to me like 
> something pip should not need to know...
> 


Differences between what files are in a sdist and what files are in a VCS 
directory will lead to different behaviors on install which makes ``pip install 
.`` and ``build-a-sdist && pip install the-sdist.tar.gz`` behave differently. 
Attempting to funnel everything through the same VCS -> sdist -> wheel path 
makes it less likely for these kinds of issues to occur.

This is not a fully resolvable problem, and it is going to happen basically 
anytime you have two independent lists of what files get put into a sdist and 
what files get installed. It is not unique to setuptools nor is it a result of 
the way distutils/setuptools works— they only expose it more obviously because 
of their relevant APIs. Infact, both enscons and flit have this same problem 
(although flit has gone to some length to minimize the issue, so it’s somewhat 
hard, but not impossible, to actually trigger it).


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Mon, Aug 28, 2017 at 9:21 AM, Donald Stufft  wrote:

> Donald, what do you think? IIRC, you were most keen on going
> sdist->wheel where possible, and I don't think you've commented on
> Paul's suggestion yet (apologies if I've overlooked a response).
>
>
> I still think it should, and prefer pip to attempt to build a sdist prior
> to building a wheel when we’re coming from a VCS directory. I think that is
> going to be the most robust mechanism with the least amount of surprising
> behavior for end users.
>

why? even as it stands, doesn't setuptools dump everything into the build
dir anyway? The "Creating of an sdist" really seems like a build-system
problem, not a package manager problem to me.

> "when we’re coming from a VCS directory."

whether this is a VCS directory or some other source seems to me like
something pip should not need to know...

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker]

On Sun, Aug 27, 2017 at 2:59 AM, Paul Moore  wrote:

> > Suppose you
> > have a frontend like pip that when given a source directory and asked
> > to build a wheel, wants to implement that as:
> >   - build sdist
> >   - unpack sdist
> >   - build wheel from unpacked sdist
>
> ... I'd like to
> point out that in
> https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html
> I confirmed that I'm basically persuaded now that pip should just
> "trust the backend" and use build_wheel directly. So this whole debate
> isn't relevant for pip.


Nor should it be for any other system. Before I saw this post I was about
to make a point about not contorting ourselves to make things works the
same way as pip and setuptools currently work -- pip was kind of hacked
together to support setuptools, and setuptools grew kinda-organically from
distutils (while also shoehorning in other not-really-building
functionality).

So let's make the "new" API clean and adapt the current tools to that.

Specifically, I'm really confused about the sdist concept -- sure, it's a
nice idea to have a standard way to create a source distributions, and I
can see that tools that, for instance, create uploads for PyPi and the like
will want to do that, but I don't see it as an inherent part of building a
binary distribution (i.e. wheel), or building and installing a package
directly.

So:

If you want a wheel, the build system should be asked to build a wheel.
If you want an sdist, the build system should be asked to build an sdist --
and it can politely decline.

Whether the build system first makes an sdist, and then builds a wheel from
that it entirely up to the build system.

Is there any reason to make this more complicated?

We'll need to wait for other
> currently-hypothetical frontends like tox or twine which might use the
> build_wheel hook, before we'll have any practical experience to
> confirm if there's an issue with "return None" as the way for backends
> to signal that they weren't able to build a wheel.
>

Sure -- though I'm with Nathaniel on this one (I think -- kinda hard to
keep th threads straight!) getting a NOne can happen for alot of reasons,
better to have a more information-rich way to know that something failed.

And isn't "be able to build a wheel" a requirement for ANY supported
back-end? In which case, failure to build one is a failure that could have
been caused by any number of issues -- shouldn't it simply fail with an
Exception -- and the back end will hopefully generate nice exceptions for
the common failures?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Donald Stufft]

> On Aug 28, 2017, at 4:43 AM, Thomas Kluyver  wrote:
> 
> If pip does uses build_wheel directly, as Paul now prefers, I think we
> can leave the NotImplemented/Error/None question for a later date. We
> only want some way to signal "I can't do that" because a frontend might
> try sdist->wheel with a fallback to making a wheel directly. If no
> frontend is actually planning to do that, we can leave specifying it
> until a frontend wants it.
> 
> Donald, what do you think? IIRC, you were most keen on going
> sdist->wheel where possible, and I don't think you've commented on
> Paul's suggestion yet (apologies if I've overlooked a response).
> 


I still think it should, and prefer pip to attempt to build a sdist prior to 
building a wheel when we’re coming from a VCS directory. I think that is going 
to be the most robust mechanism with the least amount of surprising behavior 
for end users.

I also think it is important to define this fallback now, because doing it now 
means we have flexibility to adjust the strategy that frontends use without 
going back through the PEP process and waiting on backends to update to the 
latest version of the PEP.

I am okay with an exception if it’s a specific exception, the idea of having an 
exception class exported as part of the API is fine with me. I think it’s still 
a worse API to use an exception for a non-exceptional case, but if it’s a 
specific exception it at least prevents bubbling up unrelated errors and having 
them be treated as asking for the fallback behavior if possible.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Thomas Kluyver]

If pip does uses build_wheel directly, as Paul now prefers, I think we
can leave the NotImplemented/Error/None question for a later date. We
only want some way to signal "I can't do that" because a frontend might
try sdist->wheel with a fallback to making a wheel directly. If no
frontend is actually planning to do that, we can leave specifying it
until a frontend wants it.

Donald, what do you think? IIRC, you were most keen on going
sdist->wheel where possible, and I don't think you've commented on
Paul's suggestion yet (apologies if I've overlooked a response).

Thomas

On Mon, Aug 28, 2017, at 12:47 AM, Nathaniel Smith wrote:
> On Sun, Aug 27, 2017 at 4:27 PM, Greg Ewing 
> wrote:
> > Nathaniel Smith wrote:
> >>
> >> - creating an sdist failed for unexpected reasons, that need a human
> >> to sort out (due to a broken system, or bugs – hey, they happen – or
> >> ...)
> >
> >
> > I think that should still be reported via an exception. Returning
> > None should only be for the specific case that the backend doesn't
> > support the requested operation.
> 
> Well, you can't exactly say "if your code is buggy, then you should
> signal that by doing this well defined thing" :-). One of my
> objections to None is that it's very easy to return accidentally,
> i.e., buggy code *will* sometimes return None no matter what the spec
> says.
> 
> -n
> 
> -- 
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Sun, Aug 27, 2017 at 4:27 PM, Greg Ewing  wrote:
> Nathaniel Smith wrote:
>>
>> - creating an sdist failed for unexpected reasons, that need a human
>> to sort out (due to a broken system, or bugs – hey, they happen – or
>> ...)
>
>
> I think that should still be reported via an exception. Returning
> None should only be for the specific case that the backend doesn't
> support the requested operation.

Well, you can't exactly say "if your code is buggy, then you should
signal that by doing this well defined thing" :-). One of my
objections to None is that it's very easy to return accidentally,
i.e., buggy code *will* sometimes return None no matter what the spec
says.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Greg Ewing]

Nathaniel Smith wrote:

- creating an sdist failed for unexpected reasons, that need a human
to sort out (due to a broken system, or bugs – hey, they happen – or
...)


I think that should still be reported via an exception. Returning
None should only be for the specific case that the backend doesn't
support the requested operation.

--
Greg
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Daniel Holth]

Return none makes sense, but I don't think the outcome of that or the
"configurable src path" decisions will affect the success of the proposal.

On Sun, Aug 27, 2017, 05:59 Paul Moore  wrote:

> On 26 August 2017 at 23:13, Nathaniel Smith  wrote:
> > This whole discussion is about handling a specific case: suppose you
> > have a frontend like pip that when given a source directory and asked
> > to build a wheel, wants to implement that as:
> >   - build sdist
> >   - unpack sdist
> >   - build wheel from unpacked sdist
>
> While I can't speak for Donald or the other pip devs, I'd like to
> point out that in
> https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html
> I confirmed that I'm basically persuaded now that pip should just
> "trust the backend" and use build_wheel directly. So this whole debate
> isn't relevant for pip. We'll need to wait for other
> currently-hypothetical frontends like tox or twine which might use the
> build_wheel hook, before we'll have any practical experience to
> confirm if there's an issue with "return None" as the way for backends
> to signal that they weren't able to build a wheel.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Paul Moore]

On 26 August 2017 at 23:13, Nathaniel Smith  wrote:
> This whole discussion is about handling a specific case: suppose you
> have a frontend like pip that when given a source directory and asked
> to build a wheel, wants to implement that as:
>   - build sdist
>   - unpack sdist
>   - build wheel from unpacked sdist

While I can't speak for Donald or the other pip devs, I'd like to
point out that in
https://mail.python.org/pipermail/distutils-sig/2017-August/031288.html
I confirmed that I'm basically persuaded now that pip should just
"trust the backend" and use build_wheel directly. So this whole debate
isn't relevant for pip. We'll need to wait for other
currently-hypothetical frontends like tox or twine which might use the
build_wheel hook, before we'll have any practical experience to
confirm if there's an issue with "return None" as the way for backends
to signal that they weren't able to build a wheel.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

The difference is that an empty list for requires would indicate that
building is possible but none would not. Those two values are truthily
equivalent.

For building, the name returned indicates building was possible, but it's
not truthily equivalent to none.

On Aug 27, 2017 12:08 AM, "C Anthony Risinger" 
wrote:

> On Sat, Aug 26, 2017 at 11:18 PM, xoviat  wrote:
>
>> > and they do not appear to receive the source or wheel directories.
>>
>> The source directory is the current directory, if I am not mistaken.
>>
>
> Oh right right, for some reason I was reading the `build_*` hooks as
> receiving the input directory rather than the output directory. I don't
> see it stated anywhere that the PWD will be the source directory however.
> Might be a good addition.
>
> In this case, `get_requires_for_*` hooks should also be able to depend on
> PWD being the applicable source directory (which of course might be an
> unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel
> pipeline).
>
>
>> > This lets `build_*` focus purely on building things straight away.
>> There is a difference between "no more reqs are needed to do X" and "no
>> possible req will achieve X" even though both add zero requirements. Why
>> not let this hook relay it's decision more completely?
>>
>> Not trying to speak on behalf of flit here, but if I understand
>> correctly, flit requires git to build a source distribution. Flit knows its
>> build requirements, so it can just return them when get_requires is called.
>> However, it needs to attempt to invoke git to find out whether it can build
>> a source distribution, which if I understand correctly, is a lengthy
>> operation (whether that's actually true is not actually relevant because we
>> are discussion potentially any backend/operation). It's more efficient if
>> git is only invoked once, and if a failure occurs, then under the proposal
>> "None" would be returned from build_sdist rather than the name of the built
>> source distribution.
>>
>
> What is the benefit in making `build_sdist` handle this assertion?
> Requirements checking is only performed once, earlier than building.
> Somewhere, somehow, something must look to see if git exists. Is it not
> more appropriate to check early, while asserting python requirements, vs.
> actually being told to build an sdist? That is already the expectation on
> the wheel side of things since it would also fail if necessary system
> binaries are missing. Checking for git is conflating system reqs and python
> reqs as "requirements", but I don't see why the get_requires_* hooks can't
> do both if they so choose.
>
>
>>  def get_requires_for_build_sdist(source_dir, ...):
>> # I have no interest in sdists and I never will.
>> # GO AWAY.
>> return None
>>
>> I have to say that the above example is not a good idea. Perhaps I was a
>> bit muddled on that point earlier: the reason that "return None" is a good
>> idea for build sdist is, well:
>>
>> def build_sdist(...):  # Indicates success
>> return "x.tar.gz"
>>
>> def build_sdist(...):  # Indicates failure
>> return None
>>
>>
>> Those are not truthily equivalent, which is important because it means
>> that someone is unlikely to make a mistake on that matter.
>>
>
> What is the difference? Both equally inform the frontend that building an
> sdist is not going to happen.
>
> At the end of the day, I'm not too partial to any of the solutions I've
> read thus far, and do not have any desire to derail progress. I just don't
> like the variation in `build_sdist` vs. `build_wheel` because it feels like
> a kludge to me. I thought perhaps defaulting sdists to "unsupported" and
> requiring a backend to opt-in was both more elegant and more explicit,
> allowing tools like flit to better express their conditional sdist support.
>
> --
>
> C Anthony
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[C Anthony Risinger]

On Sat, Aug 26, 2017 at 11:18 PM, xoviat  wrote:

> > and they do not appear to receive the source or wheel directories.
>
> The source directory is the current directory, if I am not mistaken.
>

Oh right right, for some reason I was reading the `build_*` hooks as
receiving the input directory rather than the output directory. I don't see
it stated anywhere that the PWD will be the source directory however. Might
be a good addition.

In this case, `get_requires_for_*` hooks should also be able to depend on
PWD being the applicable source directory (which of course might be an
unpacked sdist like in pip's proposed multistage sdist -> unpack -> wheel
pipeline).


> > This lets `build_*` focus purely on building things straight away.
> There is a difference between "no more reqs are needed to do X" and "no
> possible req will achieve X" even though both add zero requirements. Why
> not let this hook relay it's decision more completely?
>
> Not trying to speak on behalf of flit here, but if I understand correctly,
> flit requires git to build a source distribution. Flit knows its build
> requirements, so it can just return them when get_requires is called.
> However, it needs to attempt to invoke git to find out whether it can build
> a source distribution, which if I understand correctly, is a lengthy
> operation (whether that's actually true is not actually relevant because we
> are discussion potentially any backend/operation). It's more efficient if
> git is only invoked once, and if a failure occurs, then under the proposal
> "None" would be returned from build_sdist rather than the name of the built
> source distribution.
>

What is the benefit in making `build_sdist` handle this assertion?
Requirements checking is only performed once, earlier than building.
Somewhere, somehow, something must look to see if git exists. Is it not
more appropriate to check early, while asserting python requirements, vs.
actually being told to build an sdist? That is already the expectation on
the wheel side of things since it would also fail if necessary system
binaries are missing. Checking for git is conflating system reqs and python
reqs as "requirements", but I don't see why the get_requires_* hooks can't
do both if they so choose.


>  def get_requires_for_build_sdist(source_dir, ...):
> # I have no interest in sdists and I never will.
> # GO AWAY.
> return None
>
> I have to say that the above example is not a good idea. Perhaps I was a
> bit muddled on that point earlier: the reason that "return None" is a good
> idea for build sdist is, well:
>
> def build_sdist(...):  # Indicates success
> return "x.tar.gz"
>
> def build_sdist(...):  # Indicates failure
> return None
>
>
> Those are not truthily equivalent, which is important because it means
> that someone is unlikely to make a mistake on that matter.
>

What is the difference? Both equally inform the frontend that building an
sdist is not going to happen.

At the end of the day, I'm not too partial to any of the solutions I've
read thus far, and do not have any desire to derail progress. I just don't
like the variation in `build_sdist` vs. `build_wheel` because it feels like
a kludge to me. I thought perhaps defaulting sdists to "unsupported" and
requiring a backend to opt-in was both more elegant and more explicit,
allowing tools like flit to better express their conditional sdist support.

-- 

C Anthony
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[C Anthony Risinger]

On Sat, Aug 26, 2017 at 11:05 PM, C Anthony Risinger 
wrote:

> On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith  wrote:
>
>> On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger
>>  wrote:
>> > On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:
>> >
>> > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
>> >  wrote:
>> >
>> > Sure sure, I understand all that, and why we think we need some special
>> > error signal from `build_sdist`, as currently written.
>> >
>> > What I'm suggesting, is maybe calling `build_sdist` without knowing if
>> it
>> > can succeed is already a mistake.
>> >
>> > Consider instead, if we make the following small changes:
>> >
>> > 1. `get_requires_for_build_*` is passed the sdist and wheel directories,
>> > just like `build_*`, giving them the chance to actually look at tree
>> before
>> > deciding what other reqs might be necessary.
>>
>> That's not a change, that's how it works :-).
>>
>
> Is that a change I missed from this thread? I'm reading here:
>
> https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301
> https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254
> https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist
> https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel
>
> and they do not appear to receive the source or wheel directories.
>
>
>> > 2. `get_requires_for_build_*` returns None to signal `build_*` is
>> > unsupported (superceded by static reqs defined in TOML) and [...] to
>> signal
>> > support (can be empty).
>> >
>> > 3. `get_requires_for_build_*` assumed to return None if missing (so
>> optional
>> > and implies no support).
>>
>> This is what I originally proposed, except you use None where I use
>> NotImplemented, which has the disadvantages I noted earlier. Also,
>> people didn't like the missing get_requires_for_build_* being treated
>> as no-support, which makes sense, since we expect that
>> get_requires_for_build_* won't be used very often. But one can switch
>> the default here without affecting much else. The reason we want to
>> let build_sdist report failure is just for convenience of backends who
>> don't have any other reason to implement get_requires_for_build_sdist.
>>
>
> Oh OK, good good. Well in that case I agree with you and missed the
> suggestion. I personally prefer NotImplemented as well here but None seemed
> mostly just as good and did not elicit as much pushback. It's not too big
> of deal either way.
>
> However, a missing `get_requires_for_build_wheel` technically signaling
> "unsupported" makes good sense to me because it's always supplemented by
> the static *and mandatory* `build-system.requires` list. There is no proper
> way (without breaking the spec) to signal "unsupported" for `build_wheel`
> since the backend itself (setuptools, wheel, flit) is specified here.
> "Unsupported" is only signaled when *both* the static and dynamic requires
> are None (or NotImplemented as mentioned). The kicker here in my offering,
> is that the presence of `build-system.requires` *does not in any way imply*
> `build_sdist` support. As written, there is no way to statically set the
> requirements for sdist support (though this could be changed of course with
> a new TOML key/table), so you must explicitly signal it with something like:
>
> def get_requires_for_build_sdist(*args, **kwds): return []
>
> This means, by default, `build_wheel` is "supported" and `build_sdist` is
> "unsupported", and both are no fail operations. If called, any exception is
> fatal to the entire process. If a backend goes through the work of
> supporting sdist creation, is it really a problem to relay this support
> with a 3 line function definition?
>
> Wheel (could also define nothing at all):
>
> def get_requires_for_build_sdist(source_dir, ...):
> # I have no interest in sdists and I never will.
> # GO AWAY.
> return None
>
> Flit:
>
> def get_requires_for_build_sdist(source_dir, ...):
> # I can output an sdist if the source directory is a VCS checkout I
> understand.
> requires = get_requires_for_vcs_checkout_or_signal_unsupported(source_
> dir)
> return requires
>
> Setuptools:
>
> def get_requires_for_build_sdist(source_dir, ...):
> # I'm going to successfully create an sdist or die trying!
> # There is literally no directory I can't handle so I don't even look.
> return []
>
> This seems pretty straightforward to me and avoids overloading
> `build_sdist`, keeping it no fail like `build_wheel`.
>
> Semantically, it's really the job of the requirements discovery mechanism
> to decide one of:
>
> a) Zero or more requirements are needed to transform target source_dir
> into an sdist.
> b) No requirement enables my backend to transform target source_dir into
> an sdist.
>
> This lets `build_*` focus purely on building things straight away. There
> is a difference between 

Re: [Distutils] PEP 517 again

[xoviat]

> and they do not appear to receive the source or wheel directories.

The source directory is the current directory, if I am not mistaken.

> This lets `build_*` focus purely on building things straight away. There
is a difference between "no more reqs are needed to do X" and "no possible
req will achieve X" even though both add zero requirements. Why not let
this hook relay it's decision more completely?

Not trying to speak on behalf of flit here, but if I understand correctly,
flit requires git to build a source distribution. Flit knows its build
requirements, so it can just return them when get_requires is called.
However, it needs to attempt to invoke git to find out whether it can build
a source distribution, which if I understand correctly, is a lengthy
operation (whether that's actually true is not actually relevant because we
are discussion potentially any backend/operation). It's more efficient if
git is only invoked once, and if a failure occurs, then under the proposal
"None" would be returned from build_sdist rather than the name of the built
source distribution.

 def get_requires_for_build_sdist(source_dir, ...):
# I have no interest in sdists and I never will.
# GO AWAY.
return None

I have to say that the above example is not a good idea. Perhaps I was a
bit muddled on that point earlier: the reason that "return None" is a good
idea for build sdist is, well:

def build_sdist(...):  # Indicates success
return "x.tar.gz"

def build_sdist(...):  # Indicates failure
return None


Those are not truthily equivalent, which is important because it means that
someone is unlikely to make a mistake on that matter.

2017-08-26 23:05 GMT-05:00 C Anthony Risinger :

> On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith  wrote:
>
>> On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger
>>  wrote:
>> > On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:
>> >
>> > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
>> >  wrote:
>> >
>> > Sure sure, I understand all that, and why we think we need some special
>> > error signal from `build_sdist`, as currently written.
>> >
>> > What I'm suggesting, is maybe calling `build_sdist` without knowing if
>> it
>> > can succeed is already a mistake.
>> >
>> > Consider instead, if we make the following small changes:
>> >
>> > 1. `get_requires_for_build_*` is passed the sdist and wheel directories,
>> > just like `build_*`, giving them the chance to actually look at tree
>> before
>> > deciding what other reqs might be necessary.
>>
>> That's not a change, that's how it works :-).
>>
>
> Is that a change I missed from this thread? I'm reading here:
>
> https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301
> https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254
> https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist
> https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel
>
> and they do not appear to receive the source or wheel directories.
>
>
>> > 2. `get_requires_for_build_*` returns None to signal `build_*` is
>> > unsupported (superceded by static reqs defined in TOML) and [...] to
>> signal
>> > support (can be empty).
>> >
>> > 3. `get_requires_for_build_*` assumed to return None if missing (so
>> optional
>> > and implies no support).
>>
>> This is what I originally proposed, except you use None where I use
>> NotImplemented, which has the disadvantages I noted earlier. Also,
>> people didn't like the missing get_requires_for_build_* being treated
>> as no-support, which makes sense, since we expect that
>> get_requires_for_build_* won't be used very often. But one can switch
>> the default here without affecting much else. The reason we want to
>> let build_sdist report failure is just for convenience of backends who
>> don't have any other reason to implement get_requires_for_build_sdist.
>>
>
> Oh OK, good good. Well in that case I agree with you and missed the
> suggestion. I personally prefer NotImplemented as well here but None seemed
> mostly just as good and did not elicit as much pushback. It's not too big
> of deal either way.
>
> However, a missing `get_requires_for_build_wheel` technically signaling
> "unsupported" makes good sense to me because it's always supplemented by
> the static *and mandatory* `build-system.requires` list. There is no proper
> way (without breaking the spec) to signal "unsupported" for `build_wheel`
> since the backend itself (setuptools, wheel, flit) is specified here.
> "Unsupported" is only signaled when *both* the static and dynamic requires
> are None (or NotImplemented as mentioned). The kicker here in my offering,
> is that the presence of `build-system.requires` *does not in any way imply*
> `build_sdist` support. As written, there is no way to statically set the
> requirements for sdist support (though this could be changed of course with
> a 

Re: [Distutils] PEP 517 again

[C Anthony Risinger]

On Sat, Aug 26, 2017 at 9:00 PM, Nathaniel Smith  wrote:

> On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger
>  wrote:
> > On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:
> >
> > On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
> >  wrote:
> >
> > Sure sure, I understand all that, and why we think we need some special
> > error signal from `build_sdist`, as currently written.
> >
> > What I'm suggesting, is maybe calling `build_sdist` without knowing if it
> > can succeed is already a mistake.
> >
> > Consider instead, if we make the following small changes:
> >
> > 1. `get_requires_for_build_*` is passed the sdist and wheel directories,
> > just like `build_*`, giving them the chance to actually look at tree
> before
> > deciding what other reqs might be necessary.
>
> That's not a change, that's how it works :-).
>

Is that a change I missed from this thread? I'm reading here:

https://github.com/python/peps/blob/597ffba/pep-0517.txt#L301
https://github.com/python/peps/blob/597ffba/pep-0517.txt#L254
https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-sdist
https://www.python.org/dev/peps/pep-0517/#get-requires-for-build-wheel

and they do not appear to receive the source or wheel directories.


> > 2. `get_requires_for_build_*` returns None to signal `build_*` is
> > unsupported (superceded by static reqs defined in TOML) and [...] to
> signal
> > support (can be empty).
> >
> > 3. `get_requires_for_build_*` assumed to return None if missing (so
> optional
> > and implies no support).
>
> This is what I originally proposed, except you use None where I use
> NotImplemented, which has the disadvantages I noted earlier. Also,
> people didn't like the missing get_requires_for_build_* being treated
> as no-support, which makes sense, since we expect that
> get_requires_for_build_* won't be used very often. But one can switch
> the default here without affecting much else. The reason we want to
> let build_sdist report failure is just for convenience of backends who
> don't have any other reason to implement get_requires_for_build_sdist.
>

Oh OK, good good. Well in that case I agree with you and missed the
suggestion. I personally prefer NotImplemented as well here but None seemed
mostly just as good and did not elicit as much pushback. It's not too big
of deal either way.

However, a missing `get_requires_for_build_wheel` technically signaling
"unsupported" makes good sense to me because it's always supplemented by
the static *and mandatory* `build-system.requires` list. There is no proper
way (without breaking the spec) to signal "unsupported" for `build_wheel`
since the backend itself (setuptools, wheel, flit) is specified here.
"Unsupported" is only signaled when *both* the static and dynamic requires
are None (or NotImplemented as mentioned). The kicker here in my offering,
is that the presence of `build-system.requires` *does not in any way imply*
`build_sdist` support. As written, there is no way to statically set the
requirements for sdist support (though this could be changed of course with
a new TOML key/table), so you must explicitly signal it with something like:

def get_requires_for_build_sdist(*args, **kwds): return []

This means, by default, `build_wheel` is "supported" and `build_sdist` is
"unsupported", and both are no fail operations. If called, any exception is
fatal to the entire process. If a backend goes through the work of
supporting sdist creation, is it really a problem to relay this support
with a 3 line function definition?

Wheel (could also define nothing at all):

def get_requires_for_build_sdist(source_dir, ...):
# I have no interest in sdists and I never will.
# GO AWAY.
return None

Flit:

def get_requires_for_build_sdist(source_dir, ...):
# I can output an sdist if the source directory is a VCS checkout I
understand.
requires =
get_requires_for_vcs_checkout_or_signal_unsupported(source_dir)
return requires

Setuptools:

def get_requires_for_build_sdist(source_dir, ...):
# I'm going to successfully create an sdist or die trying!
# There is literally no directory I can't handle so I don't even look.
return []

This seems pretty straightforward to me and avoids overloading
`build_sdist`, keeping it no fail like `build_wheel`.

Semantically, it's really the job of the requirements discovery mechanism
to decide one of:

a) Zero or more requirements are needed to transform target source_dir into
an sdist.
b) No requirement enables my backend to transform target source_dir into an
sdist.

This lets `build_*` focus purely on building things straight away. There is
a difference between "no more reqs are needed to do X" and "no possible req
will achieve X" even though both add zero requirements. Why not let this
hook relay it's decision more completely?


> > 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static)
> >
> > 5. 

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Sat, Aug 26, 2017 at 6:30 PM, C Anthony Risinger
 wrote:
> On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:
>
> On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
>  wrote:
>
> Sure sure, I understand all that, and why we think we need some special
> error signal from `build_sdist`, as currently written.
>
> What I'm suggesting, is maybe calling `build_sdist` without knowing if it
> can succeed is already a mistake.
>
> Consider instead, if we make the following small changes:
>
> 1. `get_requires_for_build_*` is passed the sdist and wheel directories,
> just like `build_*`, giving them the chance to actually look at tree before
> deciding what other reqs might be necessary.

That's not a change, that's how it works :-).

> 2. `get_requires_for_build_*` returns None to signal `build_*` is
> unsupported (superceded by static reqs defined in TOML) and [...] to signal
> support (can be empty).
>
> 3. `get_requires_for_build_*` assumed to return None if missing (so optional
> and implies no support).

This is what I originally proposed, except you use None where I use
NotImplemented, which has the disadvantages I noted earlier. Also,
people didn't like the missing get_requires_for_build_* being treated
as no-support, which makes sense, since we expect that
get_requires_for_build_* won't be used very often. But one can switch
the default here without affecting much else. The reason we want to
let build_sdist report failure is just for convenience of backends who
don't have any other reason to implement get_requires_for_build_sdist.

> 4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static)
>
> 5. wheel reqs = `get_requires_for_build_wheel` (dynamic) +
> `build-system.requires` (static)

build-system.requires contains the requirements that are always
installed before we even try importing the backend, so they're
available to all backend hooks equally.

> 6. If no reqs are found for sdist (no declared reqs in TOML and
> `get_requires_for_build_sdist` is missing or returns None), then
> `build_sdist` is unsupported.
>
> 7. If no reqs are found for wheel (no declared reqs in TOML and
> `get_requires_for_build_wheel` is missing or returns None), then
> `build_wheel` is unsupported. This one is a spec violation because at least
> one req is expected here (the backed itself).

The TOML requires aren't really useful as a signal about whether sdist
specifically is supported. Plus I think we probably want to leave
no-requires-in-TOML as a valid option for saying "I don't need
anything installed" (maybe because the backend is shipped inside the
source tree) rather than overloading it to have extra meanings.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[C Anthony Risinger]

On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:

On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
 wrote:
> On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:
>>
>> [removed Guido from CC]
>>
>> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
>>
>> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
>> > In pretty much any other context, if you have an operation that returns
>> > an
>> > regular value or an error value, the error value should be None.
>> > (Exceptions
>> > include e.g. returning a non-negative int or -1 for errors, or True for
>> > success and False for errors.)
>>
>> So, given that build_sdist returns the path of the newly built sdist,
>> the correct way to signal "I didn't manage to build a sdist" would be
>> to return None.
>>
>> Now that it's put this way, it seems glaringly obvious to me that this
>> is the correct thing to do.
>>
>>
>> Eh... I would really prefer something that's (a) more explicit about what
>> specifically went wrong, and (b) harder to return by accident. It's not
at
>> all obvious that if the list of requirements is 'None' that means 'this
>> build supports making sdists in general but cannot make them from this
>> source tree but might still be able to make a wheel'. And if you forget
to
>> put in a return statement, then python returns None for you, which seems
>> like it could lead to some super confusing error modes.
>
>
> Why does the frontend need to know why an sdist was not created?

This whole discussion is about handling a specific case: suppose you
have a frontend like pip that when given a source directory and asked
to build a wheel, wants to implement that as:
  - build sdist
  - unpack sdist
  - build wheel from unpacked sdist

And suppose you have a backend like flit, that can build sdists from
some source directories (e.g. VCS checkouts) but not others (e.g.
unpacked sdists). We need some way for pip and flit to negotiate that
even though pip *normally* would implement its build-a-wheel operation
by first building an sdist, in this case it's ok to silently fall back
to some other strategy (like building the wheel directly in the source
tree, or manually copying the source tree somewhere else and then
building a wheel in it).

But, we don't want this fallback behavior to hide real bugs. So if the
backend says "look, I just can't do sdists here, and that's an
expected thing, it's not something where the user needs to take any
particular action like filing a bug report or fixing their system or
anything like that, so if you have an alternative way to accomplish
what you're trying to do then you should just silently discard this
error and try that", ...cool. But if it doesn't explicitly say that,
then we don't want to silently discard the error and do something
else.

It's taken a *lot* of back and forth to reach consensus that all we
need here is some special error signal from the *_sdist operations.
Let's focus on resolving that :-)

Sure sure, I understand all that, and why we think we need some special
error signal from `build_sdist`, as currently written.

What I'm suggesting, is maybe calling `build_sdist` without knowing if it
can succeed is already a mistake.

Consider instead, if we make the following small changes:

1. `get_requires_for_build_*` is passed the sdist and wheel directories,
just like `build_*`, giving them the chance to actually look at tree before
deciding what other reqs might be necessary.

2. `get_requires_for_build_*` returns None to signal `build_*` is
unsupported (superceded by static reqs defined in TOML) and [...] to signal
support (can be empty).

3. `get_requires_for_build_*` assumed to return None if missing (so
optional and implies no support).

4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static)

5. wheel reqs = `get_requires_for_build_wheel` (dynamic) +
`build-system.requires` (static)

6. If no reqs are found for sdist (no declared reqs in TOML and
`get_requires_for_build_sdist` is missing or returns None), then
`build_sdist` is unsupported.

7. If no reqs are found for wheel (no declared reqs in TOML and
`get_requires_for_build_wheel` is missing or returns None), then
`build_wheel` is unsupported. This one is a spec violation because at least
one req is expected here (the backed itself).

This arrangement allows pip to know ahead-of-time if `build_sdist` is
appropriate. If no sdist reqs are explicitly acknowledged, then no sdist
can be created. Full stop.

Example usages:

* Backeds that only support wheel creation do not implement
`get_requires_for_build_sdist` at all.

* Backeds that conditionally support sdist creation implement
`get_requires_for_build_sdist` and return None if unsupported. This is
where flit signals "impossible" from an unpacked sdist and [...] from VCS.

* Backeds that always support sdist creation (setuptools) implement
`get_requires_for_build_sdist` and return [...] 

Re: [Distutils] PEP 517 again

[C Anthony Risinger]

On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:

On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
 wrote:
> On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:
>>
>> [removed Guido from CC]
>>
>> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
>>
>> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
>> > In pretty much any other context, if you have an operation that returns
>> > an
>> > regular value or an error value, the error value should be None.
>> > (Exceptions
>> > include e.g. returning a non-negative int or -1 for errors, or True for
>> > success and False for errors.)
>>
>> So, given that build_sdist returns the path of the newly built sdist,
>> the correct way to signal "I didn't manage to build a sdist" would be
>> to return None.
>>
>> Now that it's put this way, it seems glaringly obvious to me that this
>> is the correct thing to do.
>>
>>
>> Eh... I would really prefer something that's (a) more explicit about what
>> specifically went wrong, and (b) harder to return by accident. It's not
at
>> all obvious that if the list of requirements is 'None' that means 'this
>> build supports making sdists in general but cannot make them from this
>> source tree but might still be able to make a wheel'. And if you forget
to
>> put in a return statement, then python returns None for you, which seems
>> like it could lead to some super confusing error modes.
>
>
> Why does the frontend need to know why an sdist was not created?

This whole discussion is about handling a specific case: suppose you
have a frontend like pip that when given a source directory and asked
to build a wheel, wants to implement that as:
  - build sdist
  - unpack sdist
  - build wheel from unpacked sdist

And suppose you have a backend like flit, that can build sdists from
some source directories (e.g. VCS checkouts) but not others (e.g.
unpacked sdists). We need some way for pip and flit to negotiate that
even though pip *normally* would implement its build-a-wheel operation
by first building an sdist, in this case it's ok to silently fall back
to some other strategy (like building the wheel directly in the source
tree, or manually copying the source tree somewhere else and then
building a wheel in it).

But, we don't want this fallback behavior to hide real bugs. So if the
backend says "look, I just can't do sdists here, and that's an
expected thing, it's not something where the user needs to take any
particular action like filing a bug report or fixing their system or
anything like that, so if you have an alternative way to accomplish
what you're trying to do then you should just silently discard this
error and try that", ...cool. But if it doesn't explicitly say that,
then we don't want to silently discard the error and do something
else.

It's taken a *lot* of back and forth to reach consensus that all we
need here is some special error signal from the *_sdist operations.
Let's focus on resolving that :-)


Sure sure, I understand all that, and why we think we need some special
error signal from `build_sdist`, as currently written.

What I'm suggesting, is maybe calling `build_sdist` without knowing if it
can succeed is already a mistake.

Consider instead, if we make the following small changes:

1. `get_requires_for_build_*` is passed the sdist and wheel directories,
just like `build_*`, giving them the chance to actually look at tree before
deciding what other reqs might be necessary.

2. `get_requires_for_build_*` returns None to signal `build_*` is
unsupported (superceded by static reqs defined in TOML) and [...] to signal
support (can be empty).

3. `get_requires_for_build_*` assumed to return None if missing (so
optional and implies no support).

4. sdist reqs = `get_requires_for_build_sdist` (dynamic) + ??? (static)

5. wheel reqs = `get_requires_for_build_wheel` (dynamic) +
`build-system.requires` (static)

6. If no reqs are found for sdist (no declared reqs in TOML and
`get_requires_for_build_sdist` is missing or returns None), then
`build_sdist` is unsupported.

7. If no reqs are found for wheel (no declared reqs in TOML and
`get_requires_for_build_wheel` is missing or returns None), then
`build_wheel` is unsupported. This one is a spec violation because at least
one req is expected here (the backed itself).

This arrangement allows pip to know ahead-of-time if `build_sdist` is
appropriate. If no sdist reqs are explicitly acknowledged, then no sdist
can be created. Full stop.

Example usages:

* Backeds that only support wheel creation do not implement
`get_requires_for_build_sdist` at all.

* Backeds that conditionally support sdist creation implement
`get_requires_for_build_sdist` and return None if unsupported. This is
where flit signals "impossible" from an unpacked sdist and [...] from VCS.

* Backeds that always support sdist creation (setuptools) implement
`get_requires_for_build_sdist` and return [...] 

Re: [Distutils] PEP 517 again

[xoviat]

Nathaniel:

Just to clarify, we're talking about returning none for the build function
not the get requirements function. The get requirements function is always
expected to succeed and is optional. If while the backend is discovering
requirements it finds that it cannot build, then it can return an empty
list. It cannot however signal in the get requirements function that it
cannot build. At least with what is being proposed.

The reason that the proposal works is that the build function can never
return none on success.

On Aug 26, 2017 5:13 PM, "Nathaniel Smith"  wrote:

> On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
>  wrote:
> > On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:
> >>
> >> [removed Guido from CC]
> >>
> >> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
> >>
> >> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
> >> > In pretty much any other context, if you have an operation that
> returns
> >> > an
> >> > regular value or an error value, the error value should be None.
> >> > (Exceptions
> >> > include e.g. returning a non-negative int or -1 for errors, or True
> for
> >> > success and False for errors.)
> >>
> >> So, given that build_sdist returns the path of the newly built sdist,
> >> the correct way to signal "I didn't manage to build a sdist" would be
> >> to return None.
> >>
> >> Now that it's put this way, it seems glaringly obvious to me that this
> >> is the correct thing to do.
> >>
> >>
> >> Eh... I would really prefer something that's (a) more explicit about
> what
> >> specifically went wrong, and (b) harder to return by accident. It's not
> at
> >> all obvious that if the list of requirements is 'None' that means 'this
> >> build supports making sdists in general but cannot make them from this
> >> source tree but might still be able to make a wheel'. And if you forget
> to
> >> put in a return statement, then python returns None for you, which seems
> >> like it could lead to some super confusing error modes.
> >
> >
> > Why does the frontend need to know why an sdist was not created?
>
> This whole discussion is about handling a specific case: suppose you
> have a frontend like pip that when given a source directory and asked
> to build a wheel, wants to implement that as:
>   - build sdist
>   - unpack sdist
>   - build wheel from unpacked sdist
>
> And suppose you have a backend like flit, that can build sdists from
> some source directories (e.g. VCS checkouts) but not others (e.g.
> unpacked sdists). We need some way for pip and flit to negotiate that
> even though pip *normally* would implement its build-a-wheel operation
> by first building an sdist, in this case it's ok to silently fall back
> to some other strategy (like building the wheel directly in the source
> tree, or manually copying the source tree somewhere else and then
> building a wheel in it).
>
> But, we don't want this fallback behavior to hide real bugs. So if the
> backend says "look, I just can't do sdists here, and that's an
> expected thing, it's not something where the user needs to take any
> particular action like filing a bug report or fixing their system or
> anything like that, so if you have an alternative way to accomplish
> what you're trying to do then you should just silently discard this
> error and try that", ...cool. But if it doesn't explicitly say that,
> then we don't want to silently discard the error and do something
> else.
>
> It's taken a *lot* of back and forth to reach consensus that all we
> need here is some special error signal from the *_sdist operations.
> Let's focus on resolving that :-)
>
> > Frontend is asking the backend, given the current state of the world, to
> > either produce an sdist, or not. Sans ahead-of-time knowledge (see
> below), I
> > would expect build_sdist to make some sanity checks about the world, then
> > make a binary choice about whether sdist creation is a valid goal. If not
> > possible, return None or NotImplemented or False or dict-of-reasons or
> > whatever. Only if creation was *attempted*, and in the exceptional event
> it
> > then failed, would I expect an Exception. We don't have structured
> > exceptions sadly so they can't really carry much useful information from
> a
> > protocol perspective above and beyond a simple None or the like anyway.
> >
> > I'd personally like to see some parity between build_sdist and
> build_wheel
> > in this regard. Maybe the disconnect here is we have a way to specify
> hard
> > reqs for building a wheel, statically or dynamically, and build_wheel is
> > expected to never fail, but no way to specify hard reqs needed for
> > build_sdist, necessitating this optional signaling path?
>
> Not sure what you mean about hard reqs. The reason for the lack of
> parity is that we don't currently have any use cases where build_wheel
> is expected to fail, but this is expected in some sense (not sure what

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Sat, Aug 26, 2017 at 1:47 PM, C Anthony Risinger
 wrote:
> On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:
>>
>> [removed Guido from CC]
>>
>> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
>>
>> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
>> > In pretty much any other context, if you have an operation that returns
>> > an
>> > regular value or an error value, the error value should be None.
>> > (Exceptions
>> > include e.g. returning a non-negative int or -1 for errors, or True for
>> > success and False for errors.)
>>
>> So, given that build_sdist returns the path of the newly built sdist,
>> the correct way to signal "I didn't manage to build a sdist" would be
>> to return None.
>>
>> Now that it's put this way, it seems glaringly obvious to me that this
>> is the correct thing to do.
>>
>>
>> Eh... I would really prefer something that's (a) more explicit about what
>> specifically went wrong, and (b) harder to return by accident. It's not at
>> all obvious that if the list of requirements is 'None' that means 'this
>> build supports making sdists in general but cannot make them from this
>> source tree but might still be able to make a wheel'. And if you forget to
>> put in a return statement, then python returns None for you, which seems
>> like it could lead to some super confusing error modes.
>
>
> Why does the frontend need to know why an sdist was not created?

This whole discussion is about handling a specific case: suppose you
have a frontend like pip that when given a source directory and asked
to build a wheel, wants to implement that as:
  - build sdist
  - unpack sdist
  - build wheel from unpacked sdist

And suppose you have a backend like flit, that can build sdists from
some source directories (e.g. VCS checkouts) but not others (e.g.
unpacked sdists). We need some way for pip and flit to negotiate that
even though pip *normally* would implement its build-a-wheel operation
by first building an sdist, in this case it's ok to silently fall back
to some other strategy (like building the wheel directly in the source
tree, or manually copying the source tree somewhere else and then
building a wheel in it).

But, we don't want this fallback behavior to hide real bugs. So if the
backend says "look, I just can't do sdists here, and that's an
expected thing, it's not something where the user needs to take any
particular action like filing a bug report or fixing their system or
anything like that, so if you have an alternative way to accomplish
what you're trying to do then you should just silently discard this
error and try that", ...cool. But if it doesn't explicitly say that,
then we don't want to silently discard the error and do something
else.

It's taken a *lot* of back and forth to reach consensus that all we
need here is some special error signal from the *_sdist operations.
Let's focus on resolving that :-)

> Frontend is asking the backend, given the current state of the world, to
> either produce an sdist, or not. Sans ahead-of-time knowledge (see below), I
> would expect build_sdist to make some sanity checks about the world, then
> make a binary choice about whether sdist creation is a valid goal. If not
> possible, return None or NotImplemented or False or dict-of-reasons or
> whatever. Only if creation was *attempted*, and in the exceptional event it
> then failed, would I expect an Exception. We don't have structured
> exceptions sadly so they can't really carry much useful information from a
> protocol perspective above and beyond a simple None or the like anyway.
>
> I'd personally like to see some parity between build_sdist and build_wheel
> in this regard. Maybe the disconnect here is we have a way to specify hard
> reqs for building a wheel, statically or dynamically, and build_wheel is
> expected to never fail, but no way to specify hard reqs needed for
> build_sdist, necessitating this optional signaling path?

Not sure what you mean about hard reqs. The reason for the lack of
parity is that we don't currently have any use cases where build_wheel
is expected to fail, but this is expected in some sense (not sure what
that would even mean), and there's some fallback that the frontend may
want to invoke instead.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

With respect to this issue, if everyone told me that I was wrong then I
would say I'm obviously wrong. But some people are saying one thing and
other people are saying something else.

On Aug 26, 2017 5:03 PM, "xoviat"  wrote:

> As I said, I don't care what the particular solution is on this issue.
> However I'm simply trying to anticipate and resolve potential disagreements
> that could drag this out for a significant period of time. It is clear that
> Nick and Donald have a disagreement on this issue which is actually not in
> and of itself packaging related. The clear way to resolve this was to get a
> pronouncement.
>
> On Aug 26, 2017 4:56 PM, "Nathaniel Smith"  wrote:
>
>> On Sat, Aug 26, 2017 at 2:06 PM, xoviat  wrote:
>> > I also think that Guido pretty much ruled out Notimplemented.
>>
>> As I've said, I don't think it matters a huge deal whether we use
>> NotImplemented or not. But please don't treat Guido as some kind of
>> pronouncement generating machine where you hurl out-of-context
>> questions at him and then use his response as a club to beat down
>> discussion. It's rude to Guido, it's rude to Nick and Donald (to whom
>> Guido has explicitly delegated his BDFL authority in packaging-related
>> matters), and it's rude to everyone trying to discuss proposals on
>> their merits.
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

As I said, I don't care what the particular solution is on this issue.
However I'm simply trying to anticipate and resolve potential disagreements
that could drag this out for a significant period of time. It is clear that
Nick and Donald have a disagreement on this issue which is actually not in
and of itself packaging related. The clear way to resolve this was to get a
pronouncement.

On Aug 26, 2017 4:56 PM, "Nathaniel Smith"  wrote:

> On Sat, Aug 26, 2017 at 2:06 PM, xoviat  wrote:
> > I also think that Guido pretty much ruled out Notimplemented.
>
> As I've said, I don't think it matters a huge deal whether we use
> NotImplemented or not. But please don't treat Guido as some kind of
> pronouncement generating machine where you hurl out-of-context
> questions at him and then use his response as a club to beat down
> discussion. It's rude to Guido, it's rude to Nick and Donald (to whom
> Guido has explicitly delegated his BDFL authority in packaging-related
> matters), and it's rude to everyone trying to discuss proposals on
> their merits.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Sat, Aug 26, 2017 at 2:06 PM, xoviat  wrote:
> I also think that Guido pretty much ruled out Notimplemented.

As I've said, I don't think it matters a huge deal whether we use
NotImplemented or not. But please don't treat Guido as some kind of
pronouncement generating machine where you hurl out-of-context
questions at him and then use his response as a club to beat down
discussion. It's rude to Guido, it's rude to Nick and Donald (to whom
Guido has explicitly delegated his BDFL authority in packaging-related
matters), and it's rude to everyone trying to discuss proposals on
their merits.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Nathaniel:

We're not talking about signaling failure in get_requires* we're talking
about signaling failure in build*.

On Aug 26, 2017 3:42 PM, "Nathaniel Smith"  wrote:

> On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore  wrote:
> > On 26 August 2017 at 20:17, Nathaniel Smith  wrote:
> >> Eh... I would really prefer something that's (a) more explicit about
> what
> >> specifically went wrong, and (b) harder to return by accident. It's not
> at
> >> all obvious that if the list of requirements is 'None' that means 'this
> >> build supports making sdists in general but cannot make them from this
> >> source tree but might still be able to make a wheel'. And if you forget
> to
> >> put in a return statement, then python returns None for you, which seems
> >> like it could lead to some super confusing error modes.
> >
> > Well, we've had an extensive discussion about how frontends need to
> > trust backends to get things right. I don't really see it as
> > reasonable to now argue that backends might "forget" to return the
> > right value - they might just as well "forget" to properly isolate
> > builds...
>
> It's not about division of responsibilities, it's about handling
> errors gracefully when they happen. There are three bins:
>
> - creating an sdist succeeded
> - creating an sdist failed for expected reasons, and a clever frontend
> might be able to handle the problem automatically if it understands
> what the problem is (sdist creation isn't supported in this case) and
> understands its goals (just trying to build a wheel really, so the
> sdist isn't crucial)
> - creating an sdist failed for unexpected reasons, that need a human
> to sort out (due to a broken system, or bugs – hey, they happen – or
> ...)
>
> The whole discussion has been about how we can most reliably
> distinguish between the second and third categories, and give good
> error messages for the third category. The argument for NotImplemented
> is that it avoids cases where some internal call raises
> NotImplementedError and it "leaks out" accidentally, causing a
> unexpected error to be incorrectly treated as expected error -- we
> don't want pip to be hiding real bugs in backend code. The argument
> for NotImplementedError is that it produces better error messages on
> buggy frontends. 'return None' is kind of the worst of both worlds, in
> that it's an easy thing to return accidentally, and it gives confusing
> error messages if the frontend fails to handle it properly. (Even more
> confusing, actually, because 'NoneType object has no attribute ...' is
> even harder to track down than 'NotImplementedType object has no
> attribute ...'.)
>
> > As regards an explicit description of what went wrong, why can't we
> > just use the same reporting methods that we will for any other build
> > issue (backends simply report the problem on stdout/stderr)? I don't
> > see why the backend has to package up its error information and send
> > it to the frontend to report, when we already have a perfectly
> > effective way for backends to report errors and/or warnings to the
> > user. If you're worried that the frontend might suppress the
> > information (maybe because it's planning on falling back to a direct
> > wheel build) then isn't that just the converse - backends need to
> > trust frontends to do the right thing?
>
> What I mean is more, if you're some random user and you see this in a
> build backend, what do you guess it means?
>
>   def get_requires_for_build_sdist(config_settings=None):
>   return None
>
> Now how about these?
>
>   def get_requires_for_build_sdist(config_settings=None):
>   return NotImplemented
>
>   def get_requires_for_build_sdist(config_settings=None):
>   raise NotImplementedError
>
>   def get_requires_for_build_sdist(config_settings=None):
>   raise SdistBuildNotSupported
>
> I mean, obviously return None will work. Basically anything that's
> different from "return a list or string" will work :-). That's what
> makes this a bikeshed topic, and I still think we're mostly just
> spinning our wheels here until Nick and Donald have a chance to hash
> something out that they both can agree on. But I really don't see any
> advantages to 'return None' compared to the other options that have
> been discussed
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

I also think that Guido pretty much ruled out Notimplemented.

On Aug 26, 2017 4:04 PM, "xoviat"  wrote:

> Why does the frontend need to know why an sdist was not created?
>
> I was of the opinion that such a distinction is not necessary because
> building a source distribution doesn't take that much time. However Donald
> thought that there needed to be a distinction because of the wasted time in
> attempting to build a wheel that was going to fail anyway. One of the
> things to consider is that site cythonizing takes time and maybe called for
> building source distribution. However since I think we're of the agreement
> that a source distribution should be as close to a checkout as possible,
> that may not be an issue because cythonizing may not be required to build
> the sdist.
>
> On Aug 26, 2017 3:47 PM, "C Anthony Risinger" 
> wrote:
>
>> On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:
>>
>>> [removed Guido from CC]
>>>
>>> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
>>>
>>> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
>>> > In pretty much any other context, if you have an operation that
>>> returns an
>>> > regular value or an error value, the error value should be None.
>>> (Exceptions
>>> > include e.g. returning a non-negative int or -1 for errors, or True for
>>> > success and False for errors.)
>>>
>>> So, given that build_sdist returns the path of the newly built sdist,
>>> the correct way to signal "I didn't manage to build a sdist" would be
>>> to return None.
>>>
>>> Now that it's put this way, it seems glaringly obvious to me that this
>>> is the correct thing to do.
>>>
>>>
>>> Eh... I would really prefer something that's (a) more explicit about
>>> what specifically went wrong, and (b) harder to return by accident. It's
>>> not at all obvious that if the list of requirements is 'None' that means
>>> 'this build supports making sdists in general but cannot make them from
>>> this source tree but might still be able to make a wheel'. And if you
>>> forget to put in a return statement, then python returns None for you,
>>> which seems like it could lead to some super confusing error modes.
>>>
>>
>> Why does the frontend need to know why an sdist was not created?
>>
>> Frontend is asking the backend, given the current state of the world, to
>> either produce an sdist, or not. Sans ahead-of-time knowledge (see below),
>> I would expect build_sdist to make some sanity checks about the world, then
>> make a binary choice about whether sdist creation is a valid goal. If not
>> possible, return None or NotImplemented or False or dict-of-reasons or
>> whatever. Only if creation was *attempted*, and in the exceptional event it
>> then failed, would I expect an Exception. We don't have structured
>> exceptions sadly so they can't really carry much useful information from a
>> protocol perspective above and beyond a simple None or the like anyway.
>>
>> I'd personally like to see some parity between build_sdist and
>> build_wheel in this regard. Maybe the disconnect here is we have a way to
>> specify hard reqs for building a wheel, statically or dynamically, and
>> build_wheel is expected to never fail, but no way to specify hard reqs
>> needed for build_sdist, necessitating this optional signaling path?
>>
>> If we had some definitive way for the frontend to know ahead of time if
>> build_sdist is even expected to work, it could be called with more
>> confidence.
>>
>> This could be a new sdist-related key in [build-system], a new table like
>> [sdist-system].requires, or making the get_requires_for_* less optional,
>> and defaulting to None instead of [ ].
>>
>> Frontend is responsible for prepping the world, so if it can't get a list
>> of reqs, somehow, for build_sdist, it knows it can't work. Same for
>> build_wheel, because you have to specify the backend itself, so there is at
>> least one requirement!
>>
>> Thus if you are a backend that can produce an sdist without additional
>> requirements beyond build reqs, you should explicitly return empty list
>> from get_requires_for_build_sdist.
>>
>> --
>>
>> C Anthony
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[xoviat]

Why does the frontend need to know why an sdist was not created?

I was of the opinion that such a distinction is not necessary because
building a source distribution doesn't take that much time. However Donald
thought that there needed to be a distinction because of the wasted time in
attempting to build a wheel that was going to fail anyway. One of the
things to consider is that site cythonizing takes time and maybe called for
building source distribution. However since I think we're of the agreement
that a source distribution should be as close to a checkout as possible,
that may not be an issue because cythonizing may not be required to build
the sdist.

On Aug 26, 2017 3:47 PM, "C Anthony Risinger"  wrote:

> On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:
>
>> [removed Guido from CC]
>>
>> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
>>
>> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
>> > In pretty much any other context, if you have an operation that returns
>> an
>> > regular value or an error value, the error value should be None.
>> (Exceptions
>> > include e.g. returning a non-negative int or -1 for errors, or True for
>> > success and False for errors.)
>>
>> So, given that build_sdist returns the path of the newly built sdist,
>> the correct way to signal "I didn't manage to build a sdist" would be
>> to return None.
>>
>> Now that it's put this way, it seems glaringly obvious to me that this
>> is the correct thing to do.
>>
>>
>> Eh... I would really prefer something that's (a) more explicit about what
>> specifically went wrong, and (b) harder to return by accident. It's not at
>> all obvious that if the list of requirements is 'None' that means 'this
>> build supports making sdists in general but cannot make them from this
>> source tree but might still be able to make a wheel'. And if you forget to
>> put in a return statement, then python returns None for you, which seems
>> like it could lead to some super confusing error modes.
>>
>
> Why does the frontend need to know why an sdist was not created?
>
> Frontend is asking the backend, given the current state of the world, to
> either produce an sdist, or not. Sans ahead-of-time knowledge (see below),
> I would expect build_sdist to make some sanity checks about the world, then
> make a binary choice about whether sdist creation is a valid goal. If not
> possible, return None or NotImplemented or False or dict-of-reasons or
> whatever. Only if creation was *attempted*, and in the exceptional event it
> then failed, would I expect an Exception. We don't have structured
> exceptions sadly so they can't really carry much useful information from a
> protocol perspective above and beyond a simple None or the like anyway.
>
> I'd personally like to see some parity between build_sdist and build_wheel
> in this regard. Maybe the disconnect here is we have a way to specify hard
> reqs for building a wheel, statically or dynamically, and build_wheel is
> expected to never fail, but no way to specify hard reqs needed for
> build_sdist, necessitating this optional signaling path?
>
> If we had some definitive way for the frontend to know ahead of time if
> build_sdist is even expected to work, it could be called with more
> confidence.
>
> This could be a new sdist-related key in [build-system], a new table like
> [sdist-system].requires, or making the get_requires_for_* less optional,
> and defaulting to None instead of [ ].
>
> Frontend is responsible for prepping the world, so if it can't get a list
> of reqs, somehow, for build_sdist, it knows it can't work. Same for
> build_wheel, because you have to specify the backend itself, so there is at
> least one requirement!
>
> Thus if you are a backend that can produce an sdist without additional
> requirements beyond build reqs, you should explicitly return empty list
> from get_requires_for_build_sdist.
>
> --
>
> C Anthony
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[C Anthony Risinger]

On Aug 26, 2017 2:17 PM, "Nathaniel Smith"  wrote:

> [removed Guido from CC]
>
> On Aug 26, 2017 02:29, "Paul Moore"  wrote:
>
> On 26 August 2017 at 03:17, Guido van Rossum  wrote:
> > In pretty much any other context, if you have an operation that returns
> an
> > regular value or an error value, the error value should be None.
> (Exceptions
> > include e.g. returning a non-negative int or -1 for errors, or True for
> > success and False for errors.)
>
> So, given that build_sdist returns the path of the newly built sdist,
> the correct way to signal "I didn't manage to build a sdist" would be
> to return None.
>
> Now that it's put this way, it seems glaringly obvious to me that this
> is the correct thing to do.
>
>
> Eh... I would really prefer something that's (a) more explicit about what
> specifically went wrong, and (b) harder to return by accident. It's not at
> all obvious that if the list of requirements is 'None' that means 'this
> build supports making sdists in general but cannot make them from this
> source tree but might still be able to make a wheel'. And if you forget to
> put in a return statement, then python returns None for you, which seems
> like it could lead to some super confusing error modes.
>

Why does the frontend need to know why an sdist was not created?

Frontend is asking the backend, given the current state of the world, to
either produce an sdist, or not. Sans ahead-of-time knowledge (see below),
I would expect build_sdist to make some sanity checks about the world, then
make a binary choice about whether sdist creation is a valid goal. If not
possible, return None or NotImplemented or False or dict-of-reasons or
whatever. Only if creation was *attempted*, and in the exceptional event it
then failed, would I expect an Exception. We don't have structured
exceptions sadly so they can't really carry much useful information from a
protocol perspective above and beyond a simple None or the like anyway.

I'd personally like to see some parity between build_sdist and build_wheel
in this regard. Maybe the disconnect here is we have a way to specify hard
reqs for building a wheel, statically or dynamically, and build_wheel is
expected to never fail, but no way to specify hard reqs needed for
build_sdist, necessitating this optional signaling path?

If we had some definitive way for the frontend to know ahead of time if
build_sdist is even expected to work, it could be called with more
confidence.

This could be a new sdist-related key in [build-system], a new table like
[sdist-system].requires, or making the get_requires_for_* less optional,
and defaulting to None instead of [ ].

Frontend is responsible for prepping the world, so if it can't get a list
of reqs, somehow, for build_sdist, it knows it can't work. Same for
build_wheel, because you have to specify the backend itself, so there is at
least one requirement!

Thus if you are a backend that can produce an sdist without additional
requirements beyond build reqs, you should explicitly return empty list
from get_requires_for_build_sdist.

-- 

C Anthony
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Nathaniel Smith]

On Sat, Aug 26, 2017 at 12:54 PM, Paul Moore  wrote:
> On 26 August 2017 at 20:17, Nathaniel Smith  wrote:
>> Eh... I would really prefer something that's (a) more explicit about what
>> specifically went wrong, and (b) harder to return by accident. It's not at
>> all obvious that if the list of requirements is 'None' that means 'this
>> build supports making sdists in general but cannot make them from this
>> source tree but might still be able to make a wheel'. And if you forget to
>> put in a return statement, then python returns None for you, which seems
>> like it could lead to some super confusing error modes.
>
> Well, we've had an extensive discussion about how frontends need to
> trust backends to get things right. I don't really see it as
> reasonable to now argue that backends might "forget" to return the
> right value - they might just as well "forget" to properly isolate
> builds...

It's not about division of responsibilities, it's about handling
errors gracefully when they happen. There are three bins:

- creating an sdist succeeded
- creating an sdist failed for expected reasons, and a clever frontend
might be able to handle the problem automatically if it understands
what the problem is (sdist creation isn't supported in this case) and
understands its goals (just trying to build a wheel really, so the
sdist isn't crucial)
- creating an sdist failed for unexpected reasons, that need a human
to sort out (due to a broken system, or bugs – hey, they happen – or
...)

The whole discussion has been about how we can most reliably
distinguish between the second and third categories, and give good
error messages for the third category. The argument for NotImplemented
is that it avoids cases where some internal call raises
NotImplementedError and it "leaks out" accidentally, causing a
unexpected error to be incorrectly treated as expected error -- we
don't want pip to be hiding real bugs in backend code. The argument
for NotImplementedError is that it produces better error messages on
buggy frontends. 'return None' is kind of the worst of both worlds, in
that it's an easy thing to return accidentally, and it gives confusing
error messages if the frontend fails to handle it properly. (Even more
confusing, actually, because 'NoneType object has no attribute ...' is
even harder to track down than 'NotImplementedType object has no
attribute ...'.)

> As regards an explicit description of what went wrong, why can't we
> just use the same reporting methods that we will for any other build
> issue (backends simply report the problem on stdout/stderr)? I don't
> see why the backend has to package up its error information and send
> it to the frontend to report, when we already have a perfectly
> effective way for backends to report errors and/or warnings to the
> user. If you're worried that the frontend might suppress the
> information (maybe because it's planning on falling back to a direct
> wheel build) then isn't that just the converse - backends need to
> trust frontends to do the right thing?

What I mean is more, if you're some random user and you see this in a
build backend, what do you guess it means?

  def get_requires_for_build_sdist(config_settings=None):
  return None

Now how about these?

  def get_requires_for_build_sdist(config_settings=None):
  return NotImplemented

  def get_requires_for_build_sdist(config_settings=None):
  raise NotImplementedError

  def get_requires_for_build_sdist(config_settings=None):
  raise SdistBuildNotSupported

I mean, obviously return None will work. Basically anything that's
different from "return a list or string" will work :-). That's what
makes this a bikeshed topic, and I still think we're mostly just
spinning our wheels here until Nick and Donald have a chance to hash
something out that they both can agree on. But I really don't see any
advantages to 'return None' compared to the other options that have
been discussed

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig