Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Sylvain Corlay]

Hi All,

The beta deadline for new features is approaching dangerously.

I agree with Thomas that being able to require Python 3.6 for a project
does not appear so distant for me (as soon as it is a Python 3 project
only).

Any chance to get this through? Checking support for language features will
be more and more required since new version of the C++ standard are
becoming more frequent. I understand that it is not an issue for a project
like numpy, but this is a check I do in every single one of my extension
projects.

Thanks,

Sylvain

On Thu, Aug 25, 2016 at 6:41 AM, Ralf Gommers 
wrote:

>
> On Mon, Aug 22, 2016 at 6:50 PM, Thomas Kluyver 
> wrote:
>
>> There's obviously some cost in code duplication; I haven't looked at the
>> code in question, so I don't know how bad this is.
>>
>
> This patch is pretty short and understandable, so not bad.
>
>
>> I've run into this argument before when trying to change things in
>> non-packaging-related parts of the stdlib, and I agree with Sylvain that
>> it's fundamentally problematic. If we're trying to improve the stdlib,
>> we're obviously taking a long view, but that's how we ensure the stdlib is
>> still useful in a few years time. This goes for packaging tools as much as
>> anything else.
>>
>
> This I don't agree with - packaging is fundamentally different for the
> reasons Donald gave.
>
> Ralf
>
>
>> I already have projects where I'm happy to require Python >=3.4, so being
>> able to depend on Python 3.6 is not such a distant prospect.
>>
>> Thomas
>>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Ralf Gommers]

On Mon, Aug 22, 2016 at 6:50 PM, Thomas Kluyver 
wrote:

> On Mon, Aug 22, 2016, at 07:15 AM, Sylvain Corlay wrote:
>
> I find this worrying that the main arguments to not include a patch would
> be that
>
>  - this part of the standard library is not very maintained (things don't
> get merged)
>  - earlier versions of won't have it
>
>
> Would it make sense to add it to both distutils and setuptools? The
> standard library continues to evolve, projects that require Python 3.6
> wouldn't need to use setuptools, but we could start using it sooner.
>

I don't have a problem with this, at least it avoids the main issues I
pointed out. Although I also don't see much benefit of adding the code to
distutils as well, given that the non-setuptools use is effectively
deprecated (by not adding support for new PEPs in distutils for example)
and less and less relevant every year.


> There's obviously some cost in code duplication; I haven't looked at the
> code in question, so I don't know how bad this is.
>

This patch is pretty short and understandable, so not bad.


> I've run into this argument before when trying to change things in
> non-packaging-related parts of the stdlib, and I agree with Sylvain that
> it's fundamentally problematic. If we're trying to improve the stdlib,
> we're obviously taking a long view, but that's how we ensure the stdlib is
> still useful in a few years time. This goes for packaging tools as much as
> anything else.
>

This I don't agree with - packaging is fundamentally different for the
reasons Donald gave.

Ralf


> I already have projects where I'm happy to require Python >=3.4, so being
> able to depend on Python 3.6 is not such a distant prospect.
>
> Thomas
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Sylvain Corlay]

Hi,
Any feedback on this?
Sylvain

On Mon, Aug 22, 2016 at 10:25 AM, Sylvain Corlay 
wrote:

> I think that Thomas' proposal makes sense.
>
> I would be ok to also add it to setuptools so that it can be used sooner
> by projects that don't require python 3.6.
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Sylvain Corlay]

I think that Thomas' proposal makes sense.

I would be ok to also add it to setuptools so that it can be used sooner by
projects that don't require python 3.6.

Sylvain

On Mon, Aug 22, 2016 at 9:23 AM, <tritium-l...@sdamon.com> wrote:

> Changing packaging by updating the standard library will fail.  It’s been
> attempted.
>
>
>
> The inherent problem is you need to fix packaging for people already using
> python, which means if you add a feature to the standard library, only the
> people who always run the latest and greatest can ever use the feature.  In
> a world where we are talking about Python 3.6/3.7 and python 2.7 is by far
> the most used version of python (and python usage is split pretty evenly
> between 3.4 and 3.5 IIRC), no one will use new packaging features in the
> standard library.
>
>
>
> Putting something in setuptools means it will actually be used.  I think
> some of the goals of this sig is to be able to completely sunset distutils
> and replace it with much better solutions (plural) that all speak the same
> protocol.
>
>
>
> *From:* Distutils-SIG [mailto:distutils-sig-bounces+tritium-list=
> sdamon@python.org] *On Behalf Of *Sylvain Corlay
> *Sent:* Monday, August 22, 2016 2:16 AM
> *To:* Ralf Gommers <ralf.gomm...@gmail.com>
> *Cc:* distutils-sig <distutils-sig@python.org>
> *Subject:* Re: [Distutils] Proposed new Distutils API for compiler flag
> detection (Issue26689)
>
>
>
> Hi,
>
> On Sun, Aug 21, 2016 at 10:31 PM, Ralf Gommers <ralf.gomm...@gmail.com
> > wrote:
>
>
> On top of that there are technical reasons (don't want to test
> combinations of python + setuptools that both change per release) and
> organizational ones (distutils maintenance is terrible, many simple bugfix
> patches don't get merged for ages, setuptools at least fixes regressions
> quite fast).
>
>
>
> I'm not sure if there's an official policy on adding new things to
> distutils, but if not then this request is a good time to make one.
> Assuming of course that the setuptools devs are willing to merge features
> like the one from Sylvain.
>
>
>
>
>
> I find this worrying that the main arguments to not include a patch would
> be that
>
>
>
>  - this part of the standard library is not very maintained (things don't
> get merged)
>
>  - earlier versions of won't have it
>
>
>
> The former is a bad sign for a standard library and the latter is inherent
> to any new feature. Whether it is a policy or not to not add new features
> to distutils it is clear that a code base that does not evolve is dead.
>
>
>
> How about, instead, we continue improving it?
>
>
>
> Sylvain
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[tritium-list]

Changing packaging by updating the standard library will fail.  It’s been 
attempted.

 

The inherent problem is you need to fix packaging for people already using 
python, which means if you add a feature to the standard library, only the 
people who always run the latest and greatest can ever use the feature.  In a 
world where we are talking about Python 3.6/3.7 and python 2.7 is by far the 
most used version of python (and python usage is split pretty evenly between 
3.4 and 3.5 IIRC), no one will use new packaging features in the standard 
library.

 

Putting something in setuptools means it will actually be used.  I think some 
of the goals of this sig is to be able to completely sunset distutils and 
replace it with much better solutions (plural) that all speak the same protocol.

 

From: Distutils-SIG 
[mailto:distutils-sig-bounces+tritium-list=sdamon@python.org] On Behalf Of 
Sylvain Corlay
Sent: Monday, August 22, 2016 2:16 AM
To: Ralf Gommers <ralf.gomm...@gmail.com>
Cc: distutils-sig <distutils-sig@python.org>
Subject: Re: [Distutils] Proposed new Distutils API for compiler flag detection 
(Issue26689)

 

Hi,

On Sun, Aug 21, 2016 at 10:31 PM, Ralf Gommers <ralf.gomm...@gmail.com 
<mailto:ralf.gomm...@gmail.com> > wrote:


On top of that there are technical reasons (don't want to test combinations of 
python + setuptools that both change per release) and organizational ones 
(distutils maintenance is terrible, many simple bugfix patches don't get merged 
for ages, setuptools at least fixes regressions quite fast).

 

I'm not sure if there's an official policy on adding new things to distutils, 
but if not then this request is a good time to make one. Assuming of course 
that the setuptools devs are willing to merge features like the one from 
Sylvain.

 

 

I find this worrying that the main arguments to not include a patch would be 
that

 

 - this part of the standard library is not very maintained (things don't get 
merged)

 - earlier versions of won't have it

 

The former is a bad sign for a standard library and the latter is inherent to 
any new feature. Whether it is a policy or not to not add new features to 
distutils it is clear that a code base that does not evolve is dead.

 

How about, instead, we continue improving it?

 

Sylvain

 

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Thomas Kluyver]

On Mon, Aug 22, 2016, at 07:15 AM, Sylvain Corlay wrote:
> I find this worrying that the main arguments to not include a patch
> would be that
>
>  - this part of the standard library is not very maintained (things
>don't get merged)
>  - earlier versions of won't have it

Would it make sense to add it to both distutils and setuptools? The
standard library continues to evolve, projects that require Python 3.6
wouldn't need to use setuptools, but we could start using it sooner.
There's obviously some cost in code duplication; I haven't looked at the
code in question, so I don't know how bad this is.

I've run into this argument before when trying to change things in 
non-packaging-
related parts of the stdlib, and I agree with Sylvain that it's
fundamentally problematic. If we're trying to improve the stdlib, we're
obviously taking a long view, but that's how we ensure the stdlib is
still useful in a few years time. This goes for packaging tools as much
as anything else.

I already have projects where I'm happy to require Python >=3.4, so
being able to depend on Python 3.6 is not such a distant prospect.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Sylvain Corlay]

Hi,

On Sun, Aug 21, 2016 at 10:31 PM, Ralf Gommers 
 wrote:

>
> On top of that there are technical reasons (don't want to test
> combinations of python + setuptools that both change per release) and
> organizational ones (distutils maintenance is terrible, many simple bugfix
> patches don't get merged for ages, setuptools at least fixes regressions
> quite fast).
>
> I'm not sure if there's an official policy on adding new things to
> distutils, but if not then this request is a good time to make one.
> Assuming of course that the setuptools devs are willing to merge features
> like the one from Sylvain.
>
>
I find this worrying that the main arguments to not include a patch would
be that

 - this part of the standard library is not very maintained (things don't
get merged)
 - earlier versions of won't have it

The former is a bad sign for a standard library and the latter is inherent
to any new feature. Whether it is a policy or not to not add new features
to distutils it is clear that a code base that does not evolve is dead.

How about, instead, we continue improving it?

Sylvain
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Ralf Gommers]

On Mon, Aug 22, 2016 at 6:10 AM, Donald Stufft  wrote:

> We’re reaching a point where *some* projects are announcing the end of
> Python 2 support as of a certain date, but let us not forget that Python
> 2.7 is still an order of magnitude more than any other version of Python in
> terms of downloads from PyPI.
>

Even in 5 years from now, when almost all projects have dropped support for
Python 2.7, the reasoning remains the same. Projects will then support 3 or
4 Python 3.x versions, so still any new API added to distutils cannot be
used by those projects for 3-4 years. It does not make much sense to add
new things to distutils, with the exception maybe of something needed
particularly for a new Python version (like MSVC 2015 support in Python
3.5).

On top of that there are technical reasons (don't want to test combinations
of python+setuptools that both change per release) and organizational ones
(distutils maintenance is terrible, many simple bugfix patches don't get
merged for ages, setuptools at least fixes regressions quite fast).

I'm not sure if there's an official policy on adding new things to
distutils, but if not then this request is a good time to make one.
Assuming of course that the setuptools devs are willing to merge features
like the one from Sylvain.

Ralf



>
> On Aug 21, 2016, at 2:08 PM, Sylvain Corlay 
> wrote:
>
> Although we are reaching a tipping point where a lot of projects are
> announcing the end of Python 2 support as of a certain date.
>
> Whatever is in the latest version of Python 3 when it will be considered a
> sane decision to have a Python 3-only library will be considered standard.
>
> On Sun, Aug 21, 2016 at 5:28 PM, Donald Stufft  wrote:
>
>>
>> On Aug 21, 2016, at 5:18 AM, Sylvain Corlay 
>> wrote:
>>
>> With this reasoning, nothing should ever be added to the standard library.
>>
>>
>>
>> Packaging is a bit different than other things because the network effect
>> is much more prominent. There’s no real way to say, install a backport if
>> you need one, you just have to kind of wait until every has upgraded which
>> is unlikely other bits of the standard library. In addition, people writing
>> projects in Python that are designed to be distributed, they tend to need
>> to work across many versions of Python, while someone writing a project for
>> themselves only need to worry about whatever version of Python they are
>> deploying to. So while the new statistics module is, even without a
>> backport, immediately useful to people developing their own projects for a
>> recent version of Python, something in distutils is not useful for package
>> authors until it is the *minimum* version of Python they support.
>>
>> This generally makes the reward for changing distutils very small,
>> particularly with the 3.x split because very few authors are willing to
>> drop 2.7 much less go straight to 3.6 (or whatever) and for people making
>> their own, internal projects, distutils isn’t generally used a whole lot
>> there either.
>>
>> —
>> Donald Stufft
>>
>>
>>
>>
>
>
> —
> Donald Stufft
>
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Donald Stufft]

We’re reaching a point where *some* projects are announcing the end of Python 2 
support as of a certain date, but let us not forget that Python 2.7 is still an 
order of magnitude more than any other version of Python in terms of downloads 
from PyPI.

> On Aug 21, 2016, at 2:08 PM, Sylvain Corlay  wrote:
> 
> Although we are reaching a tipping point where a lot of projects are 
> announcing the end of Python 2 support as of a certain date.
> 
> Whatever is in the latest version of Python 3 when it will be considered a 
> sane decision to have a Python 3-only library will be considered standard.
> 
> On Sun, Aug 21, 2016 at 5:28 PM, Donald Stufft  > wrote:
> 
>> On Aug 21, 2016, at 5:18 AM, Sylvain Corlay > > wrote:
>> 
>> With this reasoning, nothing should ever be added to the standard library.
> 
> 
> Packaging is a bit different than other things because the network effect is 
> much more prominent. There’s no real way to say, install a backport if you 
> need one, you just have to kind of wait until every has upgraded which is 
> unlikely other bits of the standard library. In addition, people writing 
> projects in Python that are designed to be distributed, they tend to need to 
> work across many versions of Python, while someone writing a project for 
> themselves only need to worry about whatever version of Python they are 
> deploying to. So while the new statistics module is, even without a backport, 
> immediately useful to people developing their own projects for a recent 
> version of Python, something in distutils is not useful for package authors 
> until it is the *minimum* version of Python they support.
> 
> This generally makes the reward for changing distutils very small, 
> particularly with the 3.x split because very few authors are willing to drop 
> 2.7 much less go straight to 3.6 (or whatever) and for people making their 
> own, internal projects, distutils isn’t generally used a whole lot there 
> either.
> 
> —
> Donald Stufft
> 
> 
> 
> 


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Sylvain Corlay]

Although we are reaching a tipping point where a lot of projects are
announcing the end of Python 2 support as of a certain date.

Whatever is in the latest version of Python 3 when it will be considered a
sane decision to have a Python 3-only library will be considered standard.

On Sun, Aug 21, 2016 at 5:28 PM, Donald Stufft  wrote:

>
> On Aug 21, 2016, at 5:18 AM, Sylvain Corlay 
> wrote:
>
> With this reasoning, nothing should ever be added to the standard library.
>
>
>
> Packaging is a bit different than other things because the network effect
> is much more prominent. There’s no real way to say, install a backport if
> you need one, you just have to kind of wait until every has upgraded which
> is unlikely other bits of the standard library. In addition, people writing
> projects in Python that are designed to be distributed, they tend to need
> to work across many versions of Python, while someone writing a project for
> themselves only need to worry about whatever version of Python they are
> deploying to. So while the new statistics module is, even without a
> backport, immediately useful to people developing their own projects for a
> recent version of Python, something in distutils is not useful for package
> authors until it is the *minimum* version of Python they support.
>
> This generally makes the reward for changing distutils very small,
> particularly with the 3.x split because very few authors are willing to
> drop 2.7 much less go straight to 3.6 (or whatever) and for people making
> their own, internal projects, distutils isn’t generally used a whole lot
> there either.
>
> —
> Donald Stufft
>
>
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Donald Stufft]

> On Aug 21, 2016, at 5:18 AM, Sylvain Corlay  wrote:
> 
> With this reasoning, nothing should ever be added to the standard library.


Packaging is a bit different than other things because the network effect is 
much more prominent. There’s no real way to say, install a backport if you need 
one, you just have to kind of wait until every has upgraded which is unlikely 
other bits of the standard library. In addition, people writing projects in 
Python that are designed to be distributed, they tend to need to work across 
many versions of Python, while someone writing a project for themselves only 
need to worry about whatever version of Python they are deploying to. So while 
the new statistics module is, even without a backport, immediately useful to 
people developing their own projects for a recent version of Python, something 
in distutils is not useful for package authors until it is the *minimum* 
version of Python they support.

This generally makes the reward for changing distutils very small, particularly 
with the 3.x split because very few authors are willing to drop 2.7 much less 
go straight to 3.6 (or whatever) and for people making their own, internal 
projects, distutils isn’t generally used a whole lot there either.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Sylvain Corlay]

On Sun, Aug 21, 2016 at 8:40 AM, Nathaniel Smith  wrote:

> > Having the `has_flag` in a different location from `has_function` would
> be weird in my opinion.
>
> I think the point though is that in your proposal, has_flag is in
> distutils 3.6, but has_function is in distutils 2.6, 2.7, 3.2, 3.3, 3.4,
> 3.5, 3.6. So they're still in different places in practice, and if your new
> feature is only in distutils 3.6 then almost no one will be able to use it
> within the next 3+ years. Otoh putting it in setuptools makes it uniformly
> usable for everyone immediately.
>
> -n
>

With this reasoning, nothing should ever be added to the standard library.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Nathaniel Smith]

On Aug 20, 2016 3:09 PM, "Sylvain Corlay"  wrote:
>
> Hi Ralf,
>
> On Sat, Aug 20, 2016 at 11:42 PM, Ralf Gommers 
wrote:
>>>
>>>
>>> On python-dev and in the bug tracker, Sylvain has understandably asked
for a review with an eye to adding this new feature to Python 3.6 whose
feature code cutoff is scheduled for a few weeks from now.  As release
manager, I am not opposed in general to adding new features to Distutils
but I think we should be very cautious about modifying or adding new
Distutils APIs, given that many third-party distribution authors want to
support their packages on multiple versions.  So I want to make sure that
there is some agreement that adding this new API starting with 3.6 is a
good thing to do rather than having it go in under the radar.
>>
>>
>> I'd rather see that kind of thing added to setuptools. We're already
having to deal with setuptools as a moving target, so if distutils becomes
one again as well that means more testing with combinations of different
Python and setuptools versions. Imho distutils changes should be bugfix and
essential maintenance only.
>>
>
>
> Having the `has_flag` in a different location from `has_function` would
be weird in my opinion.

I think the point though is that in your proposal, has_flag is in distutils
3.6, but has_function is in distutils 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, 3.6. So
they're still in different places in practice, and if your new feature is
only in distutils 3.6 then almost no one will be able to use it within the
next 3+ years. Otoh putting it in setuptools makes it uniformly usable for
everyone immediately.

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Sylvain Corlay]

Hi Ralf,

On Sat, Aug 20, 2016 at 11:42 PM, Ralf Gommers 
wrote:

>
>> On python-dev and in the bug tracker, Sylvain has understandably asked
>> for a review with an eye to adding this new feature to Python 3.6 whose
>> feature code cutoff is scheduled for a few weeks from now.  As release
>> manager, I am not opposed in general to adding new features to Distutils
>> but I think we should be very cautious about modifying or adding new
>> Distutils APIs, given that many third-party distribution authors want to
>> support their packages on multiple versions.  So I want to make sure that
>> there is some agreement that adding this new API starting with 3.6 is a
>> good thing to do rather than having it go in under the radar.
>
>
> I'd rather see that kind of thing added to setuptools. We're already
> having to deal with setuptools as a moving target, so if distutils becomes
> one again as well that means more testing with combinations of different
> Python and setuptools versions. Imho distutils changes should be bugfix and
> essential maintenance only.
>
>

Having the `has_flag` in a different location from `has_function` would be
weird in my opinion.

Sylvain
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Ralf Gommers]

On Sat, Aug 20, 2016 at 8:33 AM, Ned Deily  wrote:

> Some months ago, Sylvain brought up a couple of proposals for Distutils.
> The second proposal received some discussion but it appears that the first
> one got lost.  Here it is:
>
> > From sylvain.corlay at gmail.com  Wed May 25 12:01:51 2016
> > From: sylvain.corlay at gmail.com (Sylvain Corlay)
> > Date: Wed, 25 May 2016 12:01:51 -0400
> > Subject: [Distutils] Distutils improvements regarding header installation
> >  and building C extension modules
> > Message-ID: 

[Distutils] Proposed new Distutils API for compiler flag detection (Issue26689)

[Ned Deily]

Some months ago, Sylvain brought up a couple of proposals for Distutils.  The 
second proposal received some discussion but it appears that the first one got 
lost.  Here it is:

> From sylvain.corlay at gmail.com  Wed May 25 12:01:51 2016
> From: sylvain.corlay at gmail.com (Sylvain Corlay)
> Date: Wed, 25 May 2016 12:01:51 -0400
> Subject: [Distutils] Distutils improvements regarding header installation
>  and building C extension modules
> Message-ID: 
>