Re: [Distutils] What is the official position on distutils?

[Anna Ravenscroft]

+1

On Sun, Sep 4, 2016 at 7:02 AM, Steve Dower <steve.do...@python.org> wrote:

> "add it to setuptools first, and then add things to distutils where we
> feel they're sufficiently stable to not need the benefit of the faster
> setuptools update cycle"
>
> I nominate this as the official policy on API changes for distutils, with
> regular maintenance mode policies applying to anything else.
>
> Top-posted from my Windows Phone
> --
> From: Nick Coghlan <ncogh...@gmail.com>
> Sent: ‎9/‎4/‎2016 2:19
> To: Sylvain Corlay <sylvain.cor...@gmail.com>
> Cc: distutils sig <distutils-sig@python.org>
> Subject: Re: [Distutils] What is the official position on distutils?
>
> On 4 September 2016 at 06:44, Sylvain Corlay <sylvain.cor...@gmail.com>
> wrote:
> > Hi Brett,
> >
> > On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon <br...@python.org> wrote:
> >>
> >>
> >> If Jason is up for the responsibility that seems like a reasonable
> >> approach to take. It also helps test out features in setuptools first
> before
> >> upstreaming it.
> >>
> >
> > How do you see `has_flag` get into setuptools? By monkey-patching
> distutils'
> > ccompiler to have it aside of `has_function` when setuptools is imported?
> >
> > I find really weird the idea of adding this in a convoluted fashion
> instead
> > of allowing incremental improvement of distutils.
>
> The change to distutils would still be a plain patch to distutils, it
> would just be accepted at the API design level in setuptools first.
>
> The problem you're running into right now isn't a technical one - it's
> that there isn't anyone that currently feels like they have sufficient
> design authority over the distutils API to accept your proposal, hence
> Brett starting this thread to address that underlying recurring
> question, rather than the specifics of your change.
>
> Jason *definitely* has that design authority over setuptools though,
> and will be tasked with making any API additions available on older
> versions of Python via setuptools regardless of what policy we adopt
> for distutils maintenance, so if he's amenable to the idea, it makes
> sense to me to invert the order we ask those questions: add it to
> setuptools first, and then add things to distutils where we feel
> they're sufficiently stable to not need the benefit of the faster
> setuptools update cycle.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 
cordially,
Anna
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Nick Coghlan]

On 5 September 2016 at 00:27, Sylvain Corlay  wrote:
> Hi Steve,
>
> - I posted an on setuptools github
> https://github.com/pypa/setuptools/issues/771 to ask the setuptools folks
> (and especially Jason) whether they would want such a feature.
>
> I still find it baffling that the preferred way to add distutils features
> would be to monkey-patch it via setuptools first and then to add it into
> distutils later, while my proposed approach was to add it to distutils and
> monkey patch for earlier versions of python.

Imagine a world where distutils wasn't rife with global state, and
hence setuptools didn't need to monkeypatch it, and the approach seems
more logical (e.g. this is roughly the way I've worked for larger
contextlib changes - the ContextStack precursor to ExitStack only
exists in contextlib2, since I didn't push that API to the stdlib
until I was happy with both the design and the name).

> - Regarding the policy that you just nominated, does this apply to things
> that have been in pip for a long time? In the initial email that triggered
> this discussion, it was also question of adding the
> `pip.location.distutils_scheme` convenience function to distutils itself. Cf
> https://bugs.python.org/issue26955 and pip's implementation at
> https://github.com/pypa/pip/blob/281eb61b09d87765d7c2b92f6982b3fe76ccb0af/pip/locations.py#L124
>
> Indeed, the problem with certain distutils commands like `install_headers`
> is that the only way to retrieve the directory where things were installed
> is to instantiate a `distutils.dist.Distribution`
> object, which is not trivial to do. pip's distutils_scheme convenience
> function does it. This function is *about* distutils, and IMO should really
> be *in* distutils.

Should it be in distutils, or should it be in the version independent
"packaging" helper library that pip uses?

It's important to note that one of our key goals in PyPA is to get to
a point where the API of a developer's publishing toolchain of choice
only depends on the *version of the publishing tools* they're using,
and not on the specific version of Python (as long as they're using a
supported version). Encouraging the expansion and increased use of the
distutils API runs counter to that goal in most cases (since it
encourages writing publication and distribution code that will only
work with newer versions of Python), but may sometimes make sense if
it's info that really is CPython centric, or if it's a feature
primarily aimed at tightly controlled environments that can
aggressively drop support for older versions.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Sylvain Corlay]

Hi Steve,

- I posted an on setuptools github https://github.com/pypa/
setuptools/issues/771 to ask the setuptools folks (and especially Jason)
whether they would want such a feature.

I still find it baffling that the preferred way to add distutils features
would be to monkey-patch it via setuptools first and then to add it into
distutils later, while my proposed approach was to add it to distutils and
monkey patch for earlier versions of python.

- Regarding the policy that you just nominated, does this apply to things
that have been in pip for a long time? In the initial email that triggered
this discussion, it was also question of adding the
`pip.location.distutils_scheme`
convenience function to distutils itself. Cf
https://bugs.python.org/issue26955 and pip's implementation at
https://github.com/pypa/pip/blob/281eb61b09d87765d7c2b92f6982b3fe76ccb0af/pip/locations.py#L124

Indeed, the problem with certain distutils commands like `install_headers`
is that the only way to retrieve the directory where things were installed
is to instantiate a `distutils.dist.Distribution`
object, which is not trivial to do. pip's distutils_scheme convenience
function does it. This function is *about* distutils, and IMO should really
be *in* distutils.

Thanks,

Sylvain

On Sun, Sep 4, 2016 at 4:02 PM, Steve Dower <steve.do...@python.org> wrote:

> "add it to setuptools first, and then add things to distutils where we
> feel they're sufficiently stable to not need the benefit of the faster
> setuptools update cycle"
>
> I nominate this as the official policy on API changes for distutils, with
> regular maintenance mode policies applying to anything else.
>
> Top-posted from my Windows Phone
> --
> From: Nick Coghlan <ncogh...@gmail.com>
> Sent: ‎9/‎4/‎2016 2:19
> To: Sylvain Corlay <sylvain.cor...@gmail.com>
> Cc: distutils sig <distutils-sig@python.org>
> Subject: Re: [Distutils] What is the official position on distutils?
>
> On 4 September 2016 at 06:44, Sylvain Corlay <sylvain.cor...@gmail.com>
> wrote:
> > Hi Brett,
> >
> > On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon <br...@python.org> wrote:
> >>
> >>
> >> If Jason is up for the responsibility that seems like a reasonable
> >> approach to take. It also helps test out features in setuptools first
> before
> >> upstreaming it.
> >>
> >
> > How do you see `has_flag` get into setuptools? By monkey-patching
> distutils'
> > ccompiler to have it aside of `has_function` when setuptools is imported?
> >
> > I find really weird the idea of adding this in a convoluted fashion
> instead
> > of allowing incremental improvement of distutils.
>
> The change to distutils would still be a plain patch to distutils, it
> would just be accepted at the API design level in setuptools first.
>
> The problem you're running into right now isn't a technical one - it's
> that there isn't anyone that currently feels like they have sufficient
> design authority over the distutils API to accept your proposal, hence
> Brett starting this thread to address that underlying recurring
> question, rather than the specifics of your change.
>
> Jason *definitely* has that design authority over setuptools though,
> and will be tasked with making any API additions available on older
> versions of Python via setuptools regardless of what policy we adopt
> for distutils maintenance, so if he's amenable to the idea, it makes
> sense to me to invert the order we ask those questions: add it to
> setuptools first, and then add things to distutils where we feel
> they're sufficiently stable to not need the benefit of the faster
> setuptools update cycle.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Steve Dower]

"add it to setuptools first, and then add things to distutils where we feel 
they're sufficiently stable to not need the benefit of the faster setuptools 
update cycle"

I nominate this as the official policy on API changes for distutils, with 
regular maintenance mode policies applying to anything else.

Top-posted from my Windows Phone

-Original Message-
From: "Nick Coghlan" <ncogh...@gmail.com>
Sent: ‎9/‎4/‎2016 2:19
To: "Sylvain Corlay" <sylvain.cor...@gmail.com>
Cc: "distutils sig" <distutils-sig@python.org>
Subject: Re: [Distutils] What is the official position on distutils?

On 4 September 2016 at 06:44, Sylvain Corlay <sylvain.cor...@gmail.com> wrote:
> Hi Brett,
>
> On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon <br...@python.org> wrote:
>>
>>
>> If Jason is up for the responsibility that seems like a reasonable
>> approach to take. It also helps test out features in setuptools first before
>> upstreaming it.
>>
>
> How do you see `has_flag` get into setuptools? By monkey-patching distutils'
> ccompiler to have it aside of `has_function` when setuptools is imported?
>
> I find really weird the idea of adding this in a convoluted fashion instead
> of allowing incremental improvement of distutils.

The change to distutils would still be a plain patch to distutils, it
would just be accepted at the API design level in setuptools first.

The problem you're running into right now isn't a technical one - it's
that there isn't anyone that currently feels like they have sufficient
design authority over the distutils API to accept your proposal, hence
Brett starting this thread to address that underlying recurring
question, rather than the specifics of your change.

Jason *definitely* has that design authority over setuptools though,
and will be tasked with making any API additions available on older
versions of Python via setuptools regardless of what policy we adopt
for distutils maintenance, so if he's amenable to the idea, it makes
sense to me to invert the order we ask those questions: add it to
setuptools first, and then add things to distutils where we feel
they're sufficiently stable to not need the benefit of the faster
setuptools update cycle.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Nick Coghlan]

On 4 September 2016 at 06:44, Sylvain Corlay  wrote:
> Hi Brett,
>
> On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon  wrote:
>>
>>
>> If Jason is up for the responsibility that seems like a reasonable
>> approach to take. It also helps test out features in setuptools first before
>> upstreaming it.
>>
>
> How do you see `has_flag` get into setuptools? By monkey-patching distutils'
> ccompiler to have it aside of `has_function` when setuptools is imported?
>
> I find really weird the idea of adding this in a convoluted fashion instead
> of allowing incremental improvement of distutils.

The change to distutils would still be a plain patch to distutils, it
would just be accepted at the API design level in setuptools first.

The problem you're running into right now isn't a technical one - it's
that there isn't anyone that currently feels like they have sufficient
design authority over the distutils API to accept your proposal, hence
Brett starting this thread to address that underlying recurring
question, rather than the specifics of your change.

Jason *definitely* has that design authority over setuptools though,
and will be tasked with making any API additions available on older
versions of Python via setuptools regardless of what policy we adopt
for distutils maintenance, so if he's amenable to the idea, it makes
sense to me to invert the order we ask those questions: add it to
setuptools first, and then add things to distutils where we feel
they're sufficiently stable to not need the benefit of the faster
setuptools update cycle.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Ralf Gommers]

On Sat, Sep 3, 2016 at 5:06 PM, Nick Coghlan  wrote:

> On 2 September 2016 at 19:28, Paul Moore  wrote:
> > On 2 September 2016 at 09:58, Sylvain Corlay 
> wrote:
> >> My point here was that I don't think that the proposed feature has much
> to
> >> do with the concerns that were raised about distutils in general,
> unless it
> >> is decided that incremental improvements to the library even backward
> >> compatible will not be accepted anymore.
> >
> > Agreed. I think your feature is only stalled for distutils by the lack
> > of people sufficiently comfortable with the code to apply it. The
> > suggestions to add it to setuptools are more in the way of practical
> > advice on how to make the feature available, rather than expressions
> > of a policy that "we don't make changes like that in the stdlib".
>
> However, one of the other consequences of the status quo is that if
> Jason's comfortable with a change for setuptools, there's very rarely
> going to be anyone that will argue with him if he also considers it a
> suitable addition to the next version of distutils :)
>
> Since Jason's primary involvement in distutils-sig & PyPA is as the
> lead setuptools maintainer, it's easy for folks to be unaware of the
> fact that he's a distutils maintainer as well.
>
> So perhaps that's what we should adopt as the official distutils-sig
> policy? Any proposed distutils changes should *always* go into
> setuptools, as that way they're available for all currently supported
> Python versions,


and better maintained, and easier to fix if there's bugs, etc.


> and then it's up to the setuptools project to
> escalate changes or change proposals for stdlib inclusion when they
> consider that an appropriate step.
>

+1. clear and pragmatic policy.

Ralf
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Sylvain Corlay]

Hi Brett,

On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon  wrote:
>
>
> If Jason is up for the responsibility that seems like a reasonable
> approach to take. It also helps test out features in setuptools first
> before upstreaming it.
>
>
How do you see `has_flag` get into setuptools? By monkey-patching
distutils' ccompiler to have it aside of `has_function` when setuptools is
imported?

I find really weird the idea of adding this in a convoluted fashion instead
of allowing incremental improvement of distutils.

Sylvain
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Brett Cannon]

On Fri, 2 Sep 2016 at 22:06 Nick Coghlan  wrote:

> On 2 September 2016 at 19:28, Paul Moore  wrote:
> > On 2 September 2016 at 09:58, Sylvain Corlay 
> wrote:
> >> My point here was that I don't think that the proposed feature has much
> to
> >> do with the concerns that were raised about distutils in general,
> unless it
> >> is decided that incremental improvements to the library even backward
> >> compatible will not be accepted anymore.
> >
> > Agreed. I think your feature is only stalled for distutils by the lack
> > of people sufficiently comfortable with the code to apply it. The
> > suggestions to add it to setuptools are more in the way of practical
> > advice on how to make the feature available, rather than expressions
> > of a policy that "we don't make changes like that in the stdlib".
>
> However, one of the other consequences of the status quo is that if
> Jason's comfortable with a change for setuptools, there's very rarely
> going to be anyone that will argue with him if he also considers it a
> suitable addition to the next version of distutils :)
>
> Since Jason's primary involvement in distutils-sig & PyPA is as the
> lead setuptools maintainer, it's easy for folks to be unaware of the
> fact that he's a distutils maintainer as well.
>
> So perhaps that's what we should adopt as the official distutils-sig
> policy? Any proposed distutils changes should *always* go into
> setuptools, as that way they're available for all currently supported
> Python versions, and then it's up to the setuptools project to
> escalate changes or change proposals for stdlib inclusion when they
> consider that an appropriate step.
>

If Jason is up for the responsibility that seems like a reasonable approach
to take. It also helps test out features in setuptools first before
upstreaming it.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Nick Coghlan]

On 2 September 2016 at 19:28, Paul Moore  wrote:
> On 2 September 2016 at 09:58, Sylvain Corlay  wrote:
>> My point here was that I don't think that the proposed feature has much to
>> do with the concerns that were raised about distutils in general, unless it
>> is decided that incremental improvements to the library even backward
>> compatible will not be accepted anymore.
>
> Agreed. I think your feature is only stalled for distutils by the lack
> of people sufficiently comfortable with the code to apply it. The
> suggestions to add it to setuptools are more in the way of practical
> advice on how to make the feature available, rather than expressions
> of a policy that "we don't make changes like that in the stdlib".

However, one of the other consequences of the status quo is that if
Jason's comfortable with a change for setuptools, there's very rarely
going to be anyone that will argue with him if he also considers it a
suitable addition to the next version of distutils :)

Since Jason's primary involvement in distutils-sig & PyPA is as the
lead setuptools maintainer, it's easy for folks to be unaware of the
fact that he's a distutils maintainer as well.

So perhaps that's what we should adopt as the official distutils-sig
policy? Any proposed distutils changes should *always* go into
setuptools, as that way they're available for all currently supported
Python versions, and then it's up to the setuptools project to
escalate changes or change proposals for stdlib inclusion when they
consider that an appropriate step.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Paul Moore]

On 2 September 2016 at 09:58, Sylvain Corlay  wrote:
> My point here was that I don't think that the proposed feature has much to
> do with the concerns that were raised about distutils in general, unless it
> is decided that incremental improvements to the library even backward
> compatible will not be accepted anymore.

Agreed. I think your feature is only stalled for distutils by the lack
of people sufficiently comfortable with the code to apply it. The
suggestions to add it to setuptools are more in the way of practical
advice on how to make the feature available, rather than expressions
of a policy that "we don't make changes like that in the stdlib".

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Sylvain Corlay]

Hi Brett,

On Mon, Aug 29, 2016 at 9:52 PM, Brett Cannon  wrote:

>
>
>> As we discussed earlier, even though it is not a concern with C, checking
>> for availability of a compiler flag becomes crucial when building
>> extensions with C++ since new flavors of the language emerge every couple
>> of years now. It is important to be able to output meaningful error
>> messages when the compiler does not support C++[11/14/17] features if they
>> are needed for a given extension. This is a new aspect of the landscape in
>> this area.
>>
>> Finally, adding this method is a very straightforward change. `has_flag`
>> simply comes aside `has_function` as a method of ccompiler. I don't see a
>> more natural place for it. It would be a very weird design decision in my
>> opinion to not add it there, and instead to add it to distutils.ccompiler
>> by monkeypatching it in setuptools.
>>
>
> Honestly I have no comment on your feature, Sylvain, and I'm sorry your
> proposal happens to be the catalyst to this discussion. I'm just trying to
> get a general alignment from the PyPA group as the "distutils problem"
> comes up and has the same points made every time with no general decision
> on how to handle it long-term.
>
> -Brett
>
>

My point here was that I don't think that the proposed feature has much to
do with the concerns that were raised about distutils in general, unless it
is decided that incremental improvements to the library even backward
compatible will not be accepted anymore.

Sylvain
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Nick Coghlan]

On 30 August 2016 at 23:44, Paul Moore  wrote:
> On 30 August 2016 at 13:38, Antoine Pitrou  wrote:
>> On Tue, 30 Aug 2016 13:34:22 +0100
>> Paul Moore  wrote:
>>>
>>> From what I know, anyone who is actually hacking into the internal
>>> APIs to that level tends to use only distutils (probably because as
>>> you say, setuptools is even worse) and therefore falls into my
>>> category of "people affected by distutils changes who won't mind if a
>>> change is made to setuptools".
>>
>> Except when, as you point out, pip injects setuptools nilly-willy when
>> running setup.py?
>
> I'm confused. People who hack into distutils internals can't cope with
> having setuptools injected (because their hacks conflict with
> setuptools' hacks). That's my understanding of the situation, but it's
> based mainly on what I've seen of people saying "Don't break my
> APIs!!!" when distutils changes have been proposed in the past. The
> projects involved are already outside of the current
> pip/setuptools/wheel infrastructure (they have their own install
> processes, they don't host on PyPI, they are not pip-installable or
> whatever). So injecting setuptools isn't relevant, because they are
> outside that particular ecosystem.

No, this isn't the case - if you want to reliably generate wheels (as
bdist_wheel doesn't work with plain distutils), and if you want the
installdb metadata to always be generated, then you need to use pip
and its forced setuptools injection in order to get plain distutils
projects to behave the way you want.

This means that other setup.py helper projects like numpy.distutils
need to be setuptools-tolerant so they can support folks using pip
rather than calling setup.py directly.

That consequence further means that there's isn't substantially lower
risk in only changing setuptools without changing distutils - in fact,
by introducing a further unhelpful discrepancy between the two, it
increases the risk of code that works with plain distutils on 3.6+
breaking when setuptools is injected.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Antoine Pitrou]

On Tue, 30 Aug 2016 14:44:10 +0100
Paul Moore  wrote:
> On 30 August 2016 at 13:38, Antoine Pitrou  wrote:
> > On Tue, 30 Aug 2016 13:34:22 +0100
> > Paul Moore  wrote:  
> >>
> >> From what I know, anyone who is actually hacking into the internal
> >> APIs to that level tends to use only distutils (probably because as
> >> you say, setuptools is even worse) and therefore falls into my
> >> category of "people affected by distutils changes who won't mind if a
> >> change is made to setuptools".  
> >
> > Except when, as you point out, pip injects setuptools nilly-willy when
> > running setup.py?  
> 
> I'm confused. People who hack into distutils internals can't cope with
> having setuptools injected (because their hacks conflict with
> setuptools' hacks).

I'm not sure what you mean by that.  I'm sure there are situations
where it works (I can't remember for sure whether it did the last time
I did such "hacking").

The thing is, more and more problem expect packages to install with
"pip install ." and they report bugs for that (the PyPA's official
discours is of course partly responsible for this, since it's
insisting so heavily on using pip and discouraging direct calls to
setup.py).

And the day people expect all libraries to be available as wheel files,
it will probably become worse, because I don't know how to produce
wheel files using plain distutils.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Paul Moore]

On 30 August 2016 at 13:38, Antoine Pitrou  wrote:
> On Tue, 30 Aug 2016 13:34:22 +0100
> Paul Moore  wrote:
>>
>> From what I know, anyone who is actually hacking into the internal
>> APIs to that level tends to use only distutils (probably because as
>> you say, setuptools is even worse) and therefore falls into my
>> category of "people affected by distutils changes who won't mind if a
>> change is made to setuptools".
>
> Except when, as you point out, pip injects setuptools nilly-willy when
> running setup.py?

I'm confused. People who hack into distutils internals can't cope with
having setuptools injected (because their hacks conflict with
setuptools' hacks). That's my understanding of the situation, but it's
based mainly on what I've seen of people saying "Don't break my
APIs!!!" when distutils changes have been proposed in the past. The
projects involved are already outside of the current
pip/setuptools/wheel infrastructure (they have their own install
processes, they don't host on PyPI, they are not pip-installable or
whatever). So injecting setuptools isn't relevant, because they are
outside that particular ecosystem.

What I'm trying to say (badly, it seems - for which I apologise) is
that if we change setuptools, only people working in the current
ecosystem are affected, and those projects have typically been OK with
the level of change going on in setuptools. People outside of the
pip/setuptools ecosystem rely on distutils, and appear to be heavier
users of the internal distutils APIs. As a result they appear to be
typically much more conservative in their requirements (precisely
because they had to work with an undocumented internal API). So
there's a much higher barrier to changing distutils, and making
changes in setuptools instead is less likely to break existing
projects (at a cost of making such changes not accessible to
non-setuptools projects).

This is largely off-topic by now, though. I wasn't trying to derail
the discussion about policy on distutils, so I'll drop this subthread
now, with apologies to anyone who found it a distraction from the main
point.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Antoine Pitrou]

On Tue, 30 Aug 2016 13:34:22 +0100
Paul Moore  wrote:
> 
> From what I know, anyone who is actually hacking into the internal
> APIs to that level tends to use only distutils (probably because as
> you say, setuptools is even worse) and therefore falls into my
> category of "people affected by distutils changes who won't mind if a
> change is made to setuptools".

Except when, as you point out, pip injects setuptools nilly-willy when
running setup.py?

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Paul Moore]

On 30 August 2016 at 12:14, Antoine Pitrou  wrote:
> On Sun, 28 Aug 2016 23:36:43 +0100
> Paul Moore  wrote:
>> * as a result, there would have to be an *extremely* compelling reason
>> to make a change in distutils rather than in setuptools - sufficient
>> to justify the additional risk, the extra developer effort needed, and
>> the fact that any such change is only going to benefit users of newer
>> versions of Python
>
> There is a problem with this: distutils and setuptools don't document
> or specify their internal APIs.  Yet if you want to do something
> advanced, you have to hook into the command classes and either call
> their APIs or extend them.
>
> Using distutils, this is already tedious, since you have to go read its
> source code and try to figure out what happens (it's rarely obvious).
>
> Using setuptools, this is even worse, though, because setuptools builds
> on top of distutils by modifying the command classes, and so you have
> to figure out the effect that monkeypatching has on the overall
> behaviour (in addition to figure out what happens in distutils).
>
>
> I don't know if that's bothersome enough to compound the fact that
> changes made in setuptools can benefit all Python versions, but I think
> it's useful to keep it in mind.
>
>
> (that lack of programmability is really distutils' failure, of course,
> and setuptools inherited that failure by inheriting distutils'
> architecture.)

>From what I know, anyone who is actually hacking into the internal
APIs to that level tends to use only distutils (probably because as
you say, setuptools is even worse) and therefore falls into my
category of "people affected by distutils changes who won't mind if a
change is made to setuptools". Admittedly, that's a self-perpetuating
situation...

The fact remains that (as far as it's possible to tell) very few
people do dive into distutils to that level. At least, not many people
seem to have problems caused by the current "leave distutils alone,
make changes in setuptools" approach. Ideally, someone would be
working on alternative solutions to break the remaining few's tight
dependency on distutils, but with the effort they have invested in it,
there doesn't seem to be much interest in re-engineering something
that works.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Brett Cannon]

On Sun, 28 Aug 2016 at 16:55 Sylvain Corlay 
wrote:

> Distutils seems to be the de-facto standard for building extension modules
> for Python and it is used for most of the third-party extensions out there.
> I don’t think that it is reasonable to declare that it is now only meant
> for Python itself.
>

Sorry, I didn't mean to suggest distutils was suddenly going to change or
disappear, but I did want a direction/focus for what we want distutils to
become long-term as we have this discussion of what to do with distutils
any time a change is proposed. At this point distutils is practically a
black box of code as no one wants to touch out of fear of breaking the
world and that's not sustainable.


> I actually see a contradiction in pointing out some lack of involvement in
> the library to oppose well needed improvements, especially for a tool that
> has so much adoption.
>

It's not a contradiction because new features != maintenance. And the usage
of distutils makes its lack of maintenance even more scary. I guess my key
point from what I said in my previous email is that we shouldn't
perpetually ignore the fact that distutils is not to a level of support or
quality that any of us are happy with and yet so many people rely on.

For me this means we should prioritize getting our build API defined and
sdists standardized so that distutils (or a replacement) can be in the
stdlib to build Python itself only, and after that you get your build tool
from PyPI. And if that means setuptools swallows/subsumes distutils then
that's fine with me, but my key point is we are not maintaining distutils
in a way I think any of us would choose to now and so I'm trying to prod
this question to start thinking about how to fix the maintenance problem
long-term.


>
> As we discussed earlier, even though it is not a concern with C, checking
> for availability of a compiler flag becomes crucial when building
> extensions with C++ since new flavors of the language emerge every couple
> of years now. It is important to be able to output meaningful error
> messages when the compiler does not support C++[11/14/17] features if they
> are needed for a given extension. This is a new aspect of the landscape in
> this area.
>
> Finally, adding this method is a very straightforward change. `has_flag`
> simply comes aside `has_function` as a method of ccompiler. I don't see a
> more natural place for it. It would be a very weird design decision in my
> opinion to not add it there, and instead to add it to distutils.ccompiler
> by monkeypatching it in setuptools.
>

Honestly I have no comment on your feature, Sylvain, and I'm sorry your
proposal happens to be the catalyst to this discussion. I'm just trying to
get a general alignment from the PyPA group as the "distutils problem"
comes up and has the same points made every time with no general decision
on how to handle it long-term.

-Brett


>
> Thanks,
>
> Sylvain
>
> On Mon, Aug 29, 2016 at 12:36 AM, Paul Moore  wrote:
>
>> On 28 August 2016 at 18:05, Brett Cannon  wrote:
>> > The discussion of Sylvain's proposed changes to distutils suggests that
>> > there isn't a clear-cut agreement or position of this SIG -- and thus
>> Python
>> > -- on changes to distutils, its future, etc. Is there an official
>> position
>> > I'm not aware of? If not, could we get one so we know how to handle any
>> more
>> > proposed changes to distutils?
>> >
>> > For me personally (and to start this conversation if necessary), I view
>> > distutils as the package in the stdlib that handles building extension
>> > modules in the stdlib for Python. That view means that if something
>> doesn't
>> > service that explicit goal then it shouldn't go into distutils. Now
>> that we
>> > are moving down the road towards making the build step configurable I'm
>> fine
>> > with saying that distutils is not expected to work for people other than
>> > Python itself and others can use setuptools, enscons, or other projects
>> > while we continue to improve the situation to where the build system is
>> just
>> > something you specify in pyproject.toml to generate your wheel. IOW
>> > distutils is only exposed publicly because Python doesn't hide
>> anything, but
>> > making it useful for the general case of direct usage by people is a
>> > non-goal of the package.
>>
>> FWIW, my view is:
>>
>> * distutils handles building of Python packages, both in the stdlib and
>> outside
>> * in practice, setuptools is almost always used with distutils, and
>> any proposed change to distutils could be made to setuptools just as
>> easily
>> * patches for distutils typically suffer from a lack of maintainers
>> motivated to work on them
>> * there are a few developers with a strong investment in distutils,
>> who have code that is deeply entwined with the internals of distutils,
>> and changes to distutils come with a high risk of causing issues for
>> those people
>> * 

Re: [Distutils] What is the official position on distutils?

[Ned Deily]

On Aug 28, 2016, at 22:28, Nick Coghlan  wrote:
[...]
> The grey area is cases like the one Sylvain is proposing, where it's a
> straightforward API addition to distutils with no backwards
> compatibility implications, and where we (as in distutils-sig/PyPA)
> want to encourage people to write build scripts a certain way (in this
> case, failing with a useful error message when particular compiler
> features aren't available, rather than confronting end users with a
> cryptic C/C++ compiler error).
> 
> For these cases, I think the value of including the change in
> distutils isn't so much a practical software development one, as it is
> an advertising opportunity: the What's New for 3.6 can not only
> mention the addition and recommendation for use, it can point people
> to setuptools as a version independent way of accessing the same
> capability. There are still a *lot* of people out there for whom the
> standard library *is* Python, and the feature releases are one of our
> best opportunities to reach those folks with notifications that the
> world has changed around them, and they may need to adapt their
> development practices accordingly.

I rebooted the discussion of Sylvain's http://bugs.python.org/issue26689 here 
because I think decisions about Distutils API changes rightly fall under the 
purview of the PyPA / Distutils SIG.  With my 3.6 release manager hat on, I'd 
like to make sure there is agreement here before accepting this change for the 
release.  So, my understanding is that an acceptable course of action, as 
outlined by Nick, would be to see the "has_flag" method supplied by Distutils 
for 3.6 *and* by setuptools for use in older releases.  If so, I think the 
following things need to happen:

1. Someone from the PyPA (Nick?) should formally review and approve Sylvain's 
patch on the bug tracker.

2. Someone needs to open a setuptools issue and get agreement on the setuptools 
changes.

3. Someone needs to write the doc changes for 3.6, e.g. the What's New 
"advertising" and the changes to the Distutils documentation "Distributing 
Python Modules (Legacy version)" (Doc/distuils).  Anywhere else?

Is that a fair summary?

Thanks for everyone's input on this!
--Ned

--
  Ned Deily
  n...@python.org -- []

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Nick Coghlan]

On 29 August 2016 at 03:05, Brett Cannon  wrote:
> The discussion of Sylvain's proposed changes to distutils suggests that
> there isn't a clear-cut agreement or position of this SIG -- and thus Python
> -- on changes to distutils, its future, etc. Is there an official position
> I'm not aware of? If not, could we get one so we know how to handle any more
> proposed changes to distutils?

I agree it's not clear, and we should be more explicit about it.

> For me personally (and to start this conversation if necessary), I view
> distutils as the package in the stdlib that handles building extension
> modules in the stdlib for Python. That view means that if something doesn't
> service that explicit goal then it shouldn't go into distutils.

I think that's a reasonable place to draw the line for larger features
and major refactorings, especially with pip injecting setuptools for
"pip wheel" and "pip install", even if the setup.py itself only
imports distutils.

However, in cases where distutils *does* provide a particular piece of
functionality (e.g. the recent switch to emitting sdists as "tar.gz"
by default, regardless of platform), then we should maintain a
"distutils first" policy, make the change to distutils for future
versions of Python, and then backport it to earlier versions via
setuptools.

The grey area is cases like the one Sylvain is proposing, where it's a
straightforward API addition to distutils with no backwards
compatibility implications, and where we (as in distutils-sig/PyPA)
want to encourage people to write build scripts a certain way (in this
case, failing with a useful error message when particular compiler
features aren't available, rather than confronting end users with a
cryptic C/C++ compiler error).

For these cases, I think the value of including the change in
distutils isn't so much a practical software development one, as it is
an advertising opportunity: the What's New for 3.6 can not only
mention the addition and recommendation for use, it can point people
to setuptools as a version independent way of accessing the same
capability. There are still a *lot* of people out there for whom the
standard library *is* Python, and the feature releases are one of our
best opportunities to reach those folks with notifications that the
world has changed around them, and they may need to adapt their
development practices accordingly.

The backwards compatibility situation for CPython feature releases
itself also isn't as dire as it used to be - there are only a few
tightly coupled distutils extensions (with setuptools and
numpy.distutils being the most high profile ones), and it's rare these
days for us to get through a beta period without finding and resolving
any problems that come up (it's also worth remembering that the
original major problem wasn't with a feature release, it was with a
setuptools-breaking internal refactoring that landed in a maintenance
release, so there was no beta period to find and fix compatibility
problems).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Sylvain Corlay]

Distutils seems to be the de-facto standard for building extension modules
for Python and it is used for most of the third-party extensions out there.
I don’t think that it is reasonable to declare that it is now only meant
for Python itself. I actually see a contradiction in pointing out some lack
of involvement in the library to oppose well needed improvements,
especially for a tool that has so much adoption.

As we discussed earlier, even though it is not a concern with C, checking
for availability of a compiler flag becomes crucial when building
extensions with C++ since new flavors of the language emerge every couple
of years now. It is important to be able to output meaningful error
messages when the compiler does not support C++[11/14/17] features if they
are needed for a given extension. This is a new aspect of the landscape in
this area.

Finally, adding this method is a very straightforward change. `has_flag`
simply comes aside `has_function` as a method of ccompiler. I don't see a
more natural place for it. It would be a very weird design decision in my
opinion to not add it there, and instead to add it to distutils.ccompiler
by monkeypatching it in setuptools.

Thanks,

Sylvain

On Mon, Aug 29, 2016 at 12:36 AM, Paul Moore  wrote:

> On 28 August 2016 at 18:05, Brett Cannon  wrote:
> > The discussion of Sylvain's proposed changes to distutils suggests that
> > there isn't a clear-cut agreement or position of this SIG -- and thus
> Python
> > -- on changes to distutils, its future, etc. Is there an official
> position
> > I'm not aware of? If not, could we get one so we know how to handle any
> more
> > proposed changes to distutils?
> >
> > For me personally (and to start this conversation if necessary), I view
> > distutils as the package in the stdlib that handles building extension
> > modules in the stdlib for Python. That view means that if something
> doesn't
> > service that explicit goal then it shouldn't go into distutils. Now that
> we
> > are moving down the road towards making the build step configurable I'm
> fine
> > with saying that distutils is not expected to work for people other than
> > Python itself and others can use setuptools, enscons, or other projects
> > while we continue to improve the situation to where the build system is
> just
> > something you specify in pyproject.toml to generate your wheel. IOW
> > distutils is only exposed publicly because Python doesn't hide anything,
> but
> > making it useful for the general case of direct usage by people is a
> > non-goal of the package.
>
> FWIW, my view is:
>
> * distutils handles building of Python packages, both in the stdlib and
> outside
> * in practice, setuptools is almost always used with distutils, and
> any proposed change to distutils could be made to setuptools just as
> easily
> * patches for distutils typically suffer from a lack of maintainers
> motivated to work on them
> * there are a few developers with a strong investment in distutils,
> who have code that is deeply entwined with the internals of distutils,
> and changes to distutils come with a high risk of causing issues for
> those people
> * these developers *don't* use setuptools, so changes in setuptools
> don't tend to suffer from the same level of backward compatibility
> constraints
> * implementing changes in setuptools is *far* easier than doing so in
> distutils, for the above reasons
> * changes in setuptools can be used with any version of Python,
> distutils changes are only available to people with no need to support
> anything but the latest version (which in practice is a vanishingly
> small group)
> * as a result, there would have to be an *extremely* compelling reason
> to make a change in distutils rather than in setuptools - sufficient
> to justify the additional risk, the extra developer effort needed, and
> the fact that any such change is only going to benefit users of newer
> versions of Python
>
> The reality is simply that distutils is in a somewhat unhappy place,
> with too few interested maintainers and serious compatibility problems
> to deal with, but that's not exactly news to anyone. I don't think
> that a formal statement will help much. Past history has shown that
> making pronouncements like "distutils is frozen" has not helped anyone
> - so we're just trying to work with the status quo.
>
> With regard to Sylvain's proposed changes, I haven't really seen any
> arguments justifying making the change in distutils rather than in
> setuptools - the last suggestion I saw was to make the change in
> *both* places, but I don't see any particular reason (beyond abstract
> "it makes more sense to have the change in distutils" arguments) for
> not just targeting setuptools. I may be missing something (as I'm not
> personally in a position where I'd be comfortable reviewing the
> change) but it seems to me that it's a bit of a non-issue - just
> implement in setuptools and move on. The 

Re: [Distutils] What is the official position on distutils?

[Paul Moore]

On 28 August 2016 at 18:05, Brett Cannon  wrote:
> The discussion of Sylvain's proposed changes to distutils suggests that
> there isn't a clear-cut agreement or position of this SIG -- and thus Python
> -- on changes to distutils, its future, etc. Is there an official position
> I'm not aware of? If not, could we get one so we know how to handle any more
> proposed changes to distutils?
>
> For me personally (and to start this conversation if necessary), I view
> distutils as the package in the stdlib that handles building extension
> modules in the stdlib for Python. That view means that if something doesn't
> service that explicit goal then it shouldn't go into distutils. Now that we
> are moving down the road towards making the build step configurable I'm fine
> with saying that distutils is not expected to work for people other than
> Python itself and others can use setuptools, enscons, or other projects
> while we continue to improve the situation to where the build system is just
> something you specify in pyproject.toml to generate your wheel. IOW
> distutils is only exposed publicly because Python doesn't hide anything, but
> making it useful for the general case of direct usage by people is a
> non-goal of the package.

FWIW, my view is:

* distutils handles building of Python packages, both in the stdlib and outside
* in practice, setuptools is almost always used with distutils, and
any proposed change to distutils could be made to setuptools just as
easily
* patches for distutils typically suffer from a lack of maintainers
motivated to work on them
* there are a few developers with a strong investment in distutils,
who have code that is deeply entwined with the internals of distutils,
and changes to distutils come with a high risk of causing issues for
those people
* these developers *don't* use setuptools, so changes in setuptools
don't tend to suffer from the same level of backward compatibility
constraints
* implementing changes in setuptools is *far* easier than doing so in
distutils, for the above reasons
* changes in setuptools can be used with any version of Python,
distutils changes are only available to people with no need to support
anything but the latest version (which in practice is a vanishingly
small group)
* as a result, there would have to be an *extremely* compelling reason
to make a change in distutils rather than in setuptools - sufficient
to justify the additional risk, the extra developer effort needed, and
the fact that any such change is only going to benefit users of newer
versions of Python

The reality is simply that distutils is in a somewhat unhappy place,
with too few interested maintainers and serious compatibility problems
to deal with, but that's not exactly news to anyone. I don't think
that a formal statement will help much. Past history has shown that
making pronouncements like "distutils is frozen" has not helped anyone
- so we're just trying to work with the status quo.

With regard to Sylvain's proposed changes, I haven't really seen any
arguments justifying making the change in distutils rather than in
setuptools - the last suggestion I saw was to make the change in
*both* places, but I don't see any particular reason (beyond abstract
"it makes more sense to have the change in distutils" arguments) for
not just targeting setuptools. I may be missing something (as I'm not
personally in a position where I'd be comfortable reviewing the
change) but it seems to me that it's a bit of a non-issue - just
implement in setuptools and move on. The rhetoric about "this means
that distutils is dead" doesn't feel particularly helpful.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Robert Collins]

My sense of the current position is:

 - distutils exists for both Python and the broader ecosystem to build
both extension modules and package distributions
 - we recognise that its able to do anything, but many folk find it
hard to work with - or are already invested in other build tools
 - we're afraid of changing it in ways that cause incompatibilities
with e.g. the numpy eco systems heavy customisations, and we're not
sure we can recognise whether a given change is one of those
 - and we're recommending that everyone move away from directly using
distutils to using setuptoools - but setuptools *is* distutils in many
many ways, so this doesn't loosen our dependency and stability
considerations around distutils

So basically the normal 'module goes to the stdlib to freeze' effect.

I haven't read about Sylvain's proposed changes yet - ETHINGSTODO :(.

I can happily agree with a position that making distutils directly
usable is a non-goal... but insofar as making it more usable makes
setuptools more usable, that would be a goal :/.

-Rob




On 29 August 2016 at 05:05, Brett Cannon  wrote:
> The discussion of Sylvain's proposed changes to distutils suggests that
> there isn't a clear-cut agreement or position of this SIG -- and thus Python
> -- on changes to distutils, its future, etc. Is there an official position
> I'm not aware of? If not, could we get one so we know how to handle any more
> proposed changes to distutils?
>
> For me personally (and to start this conversation if necessary), I view
> distutils as the package in the stdlib that handles building extension
> modules in the stdlib for Python. That view means that if something doesn't
> service that explicit goal then it shouldn't go into distutils. Now that we
> are moving down the road towards making the build step configurable I'm fine
> with saying that distutils is not expected to work for people other than
> Python itself and others can use setuptools, enscons, or other projects
> while we continue to improve the situation to where the build system is just
> something you specify in pyproject.toml to generate your wheel. IOW
> distutils is only exposed publicly because Python doesn't hide anything, but
> making it useful for the general case of direct usage by people is a
> non-goal of the package.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig