[Numpy-discussion] Re: NEP 29 and the faster Python release cadence

[Ralf Gommers]

On Tue, May 24, 2022 at 3:24 PM Ewout ter Hoeven <
e.m.terhoe...@student.tudelft.nl> wrote:

> Personally I would be in favor of updating NEP 29 to a support timespan in
> which at most 3 (minor) Python versions are supported. The development of
> Python is still at a high pace and NumPy is a high performance library
> which thrives when be able to adopt the latest Python features and having
> clean codebase without having "if sys.version_info[:2] < (3, n):" guards
> everywhere.
>
> Especially with the developments of the faster Faster CPython project and
> the continued improvements in type hinting support, I think shortening the
> support timespan a bit is reasonable, beneficial and proportional.
>
> More important, NEP 29 is adopted by many other projects, providing a
> signal to the ecosystem that's okay to focus on the latest Python versions.
> The overlap between users using a Python version more than 2.5 years old
> and wanting the latest features and performance is probably pretty small.
> Older NumPy and other project's releases will won't disappear and be usable
> when the Python requirements are lifted.
>

I don't think this is true. These graphs show that Python 3.7 is the
version with the most downloaded wheels for, and our 1.21.x downloads are
still higher than our 1.22.x downloads - very likely because we dropped 3.7
support in 1.22.x:
https://pypistats.org/packages/numpy
https://pepy.tech/project/numpy?versions=1.22.2&versions=1.22.3&versions=1.22.4&versions=1.21.6&versions=1.20.3

I have seen problems popping up already in a few places with latest numpy
not supported what is still the most commonly used Python version (don't
have links, sorry - but they were real packaging-related issues). So I
don't think it makes sense to shorten the time window. I also don't think
there's a need to drop one version that's urgent - it's some effort and CI
time, but the balance is decent right now.

Cheers,
Ralf



>
> In my opinion a 30 month support window would provide a nice balance. Each
> Python version is supported for over 2 minor releases, with updates on the
> maintenance branches extending that with another few months. You can
> comfortably stay a full minor Python version behind year-round and still
> use the latest NumPy. For even older Python versions the old NumPy releases
> will stay available, and it allow NumPy to move on to new Python features a
> full year earlier then with a 42 month support window. Which not only
> improves feature and performance adaptation, but also lowers maintenance,
> testing, backporting and CI effort.
>
> Ewout ter Hoeven (EwoutH)
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: ralf.gomm...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NEP 29 and the faster Python release cadence

[Thomas Caswell]

To reiterate what Ralf said, the possibility of Python going to a faster
cadence was one of the things on our mind when drafting NEP 29 (see
https://numpy.org/neps/nep-0029-deprecation_policy.html#n-minor-versions-of-python
for why did not go with a fixed number of versions) because the reality of
the effort and timelines to upgrade deployments does not change with Python
release frequency.   The two things we are trying to balance here are the
needs of the projects to use the cool new stuff in Python and the needs of
our (institutional) users who are using our libraries for operations where
updating the version of Python deployed is not trivial.

There was a lot of discussion around how big the window should be with 42mo
being about in the middle of what people advocated for.  I am aware of
institutions that are on an every-other Python upgrade pattern (py37 ->
py39 -> (expected) py311) so always having at least 3 Pythons in the window
is important.  Based on what we have seen so far, I still think 42mo is a
good choice, but do not think we have seen it in operations long enough to
draw any conclusions (the downside of year-scale planning is you need to
wait years to see if it worked out like you expected!) and hence do not
think we should make any changes to the time window for another few years.
 That said, I am currently sympathetic to making the window longer and
against making it shorter.

It may be useful to have a discussion about what "support" means though
given the combinatorial rise of (Python version) x (Python implementation)
x (Operating system) x (architecture) that we are building wheels for.  I
think it would still be within the spirit of NEP 29 to run CI on CPython
for all of the supported versions (maybe across the 3 big OS's) and then
only publish wheels for the latest (2 version of Python) x (full matrix of
the rest).

Stealing some language/concepts from Microsoft (if I recall it correctly),
we should sort out which entries in that support matrix are Level 1 (CI +
wheels), Level 2 (CI), Level 3 (we test something that looks like this),
and Level 4 (your on your own, but we will take your patches to un-break
things as you send them).  We would obviously need more thought out
definitions of the levels as well.

Tom



On Wed, May 25, 2022 at 5:23 AM Ralf Gommers  wrote:

>
>
> On Tue, May 24, 2022 at 3:24 PM Ewout ter Hoeven <
> e.m.terhoe...@student.tudelft.nl> wrote:
>
>> Personally I would be in favor of updating NEP 29 to a support timespan
>> in which at most 3 (minor) Python versions are supported. The development
>> of Python is still at a high pace and NumPy is a high performance library
>> which thrives when be able to adopt the latest Python features and having
>> clean codebase without having "if sys.version_info[:2] < (3, n):" guards
>> everywhere.
>>
>> Especially with the developments of the faster Faster CPython project and
>> the continued improvements in type hinting support, I think shortening the
>> support timespan a bit is reasonable, beneficial and proportional.
>>
>> More important, NEP 29 is adopted by many other projects, providing a
>> signal to the ecosystem that's okay to focus on the latest Python versions.
>> The overlap between users using a Python version more than 2.5 years old
>> and wanting the latest features and performance is probably pretty small.
>> Older NumPy and other project's releases will won't disappear and be usable
>> when the Python requirements are lifted.
>>
>
> I don't think this is true. These graphs show that Python 3.7 is the
> version with the most downloaded wheels for, and our 1.21.x downloads are
> still higher than our 1.22.x downloads - very likely because we dropped 3.7
> support in 1.22.x:
> https://pypistats.org/packages/numpy
>
> https://pepy.tech/project/numpy?versions=1.22.2&versions=1.22.3&versions=1.22.4&versions=1.21.6&versions=1.20.3
>
> I have seen problems popping up already in a few places with latest numpy
> not supported what is still the most commonly used Python version (don't
> have links, sorry - but they were real packaging-related issues). So I
> don't think it makes sense to shorten the time window. I also don't think
> there's a need to drop one version that's urgent - it's some effort and CI
> time, but the balance is decent right now.
>
> Cheers,
> Ralf
>
>
>
>>
>> In my opinion a 30 month support window would provide a nice balance.
>> Each Python version is supported for over 2 minor releases, with updates on
>> the maintenance branches extending that with another few months. You can
>> comfortably stay a full minor Python version behind year-round and still
>> use the latest NumPy. For even older Python versions the old NumPy releases
>> will stay available, and it allow NumPy to move on to new Python features a
>> full year earlier then with a 42 month support window. Which not only
>> improves feature and performance adaptation, but also lowers maintenance,
>> testing, backporting and CI effort.

[Numpy-discussion] Re: NEP 29 and the faster Python release cadence

[Ralf Gommers]

On Wed, May 25, 2022 at 4:56 PM Thomas Caswell  wrote:

> To reiterate what Ralf said, the possibility of Python going to a faster
> cadence was one of the things on our mind when drafting NEP 29 (see
> https://numpy.org/neps/nep-0029-deprecation_policy.html#n-minor-versions-of-python
> for why did not go with a fixed number of versions) because the reality of
> the effort and timelines to upgrade deployments does not change with Python
> release frequency.   The two things we are trying to balance here are the
> needs of the projects to use the cool new stuff in Python and the needs of
> our (institutional) users who are using our libraries for operations where
> updating the version of Python deployed is not trivial.
>
> There was a lot of discussion around how big the window should be with
> 42mo being about in the middle of what people advocated for.  I am aware of
> institutions that are on an every-other Python upgrade pattern (py37 ->
> py39 -> (expected) py311) so always having at least 3 Pythons in the window
> is important.  Based on what we have seen so far, I still think 42mo is a
> good choice, but do not think we have seen it in operations long enough to
> draw any conclusions (the downside of year-scale planning is you need to
> wait years to see if it worked out like you expected!) and hence do not
> think we should make any changes to the time window for another few years.
>  That said, I am currently sympathetic to making the window longer and
> against making it shorter.
>
> It may be useful to have a discussion about what "support" means though
> given the combinatorial rise of (Python version) x (Python implementation)
> x (Operating system) x (architecture) that we are building wheels for.  I
> think it would still be within the spirit of NEP 29 to run CI on CPython
> for all of the supported versions (maybe across the 3 big OS's) and then
> only publish wheels for the latest (2 version of Python) x (full matrix of
> the rest).
>

We cannot do that (yet, at least). Failing to publish wheels for a
supported Python version on a major OS is far worse than dropping support
completely. This will remain true until the time that Pip starts defaulting
to wheels-only and never picks the sdist if there's an older release for
that platform + Python combo.

Stealing some language/concepts from Microsoft (if I recall it correctly),
> we should sort out which entries in that support matrix are Level 1 (CI +
> wheels), Level 2 (CI), Level 3 (we test something that looks like this),
> and Level 4 (your on your own, but we will take your patches to un-break
> things as you send them).  We would obviously need more thought out
> definitions of the levels as well.
>

Agreed, this would be useful to map out. Starting with what we currently
do, and not mix in any changes in our CI/wheel coverage, in order to
decouple the support model from other, more contentious proposals.

Cheers,
Ralf


> Tom
>
>
>
> On Wed, May 25, 2022 at 5:23 AM Ralf Gommers 
> wrote:
>
>>
>>
>> On Tue, May 24, 2022 at 3:24 PM Ewout ter Hoeven <
>> e.m.terhoe...@student.tudelft.nl> wrote:
>>
>>> Personally I would be in favor of updating NEP 29 to a support timespan
>>> in which at most 3 (minor) Python versions are supported. The development
>>> of Python is still at a high pace and NumPy is a high performance library
>>> which thrives when be able to adopt the latest Python features and having
>>> clean codebase without having "if sys.version_info[:2] < (3, n):" guards
>>> everywhere.
>>>
>>> Especially with the developments of the faster Faster CPython project
>>> and the continued improvements in type hinting support, I think shortening
>>> the support timespan a bit is reasonable, beneficial and proportional.
>>>
>>> More important, NEP 29 is adopted by many other projects, providing a
>>> signal to the ecosystem that's okay to focus on the latest Python versions.
>>> The overlap between users using a Python version more than 2.5 years old
>>> and wanting the latest features and performance is probably pretty small.
>>> Older NumPy and other project's releases will won't disappear and be usable
>>> when the Python requirements are lifted.
>>>
>>
>> I don't think this is true. These graphs show that Python 3.7 is the
>> version with the most downloaded wheels for, and our 1.21.x downloads are
>> still higher than our 1.22.x downloads - very likely because we dropped 3.7
>> support in 1.22.x:
>> https://pypistats.org/packages/numpy
>>
>> https://pepy.tech/project/numpy?versions=1.22.2&versions=1.22.3&versions=1.22.4&versions=1.21.6&versions=1.20.3
>>
>> I have seen problems popping up already in a few places with latest numpy
>> not supported what is still the most commonly used Python version (don't
>> have links, sorry - but they were real packaging-related issues). So I
>> don't think it makes sense to shorten the time window. I also don't think
>> there's a need to drop one version that's urgent - it's some effort and CI
>> t

[Numpy-discussion] Re: NEP 29 and the faster Python release cadence

[David Menéndez Hurtado]

On Wed, 25 May 2022, 4:54 pm Thomas Caswell,  wrote:

>
> Stealing some language/concepts from Microsoft (if I recall it correctly),
> we should sort out which entries in that support matrix are Level 1 (CI +
> wheels), Level 2 (CI), Level 3 (we test something that looks like this),
> and Level 4 (your on your own, but we will take your patches to un-break
> things as you send them).
>

As someone who knows very little about packaging, what is the rationale
behind the difference between levels 1 and 2? How much load does it add,
given that the CI is already set up and building them?

/David
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NEP 29 and the faster Python release cadence

[Aaron Meurer]

> I have seen problems popping up already in a few places with latest numpy not 
> supported what is still the most commonly used Python version (don't have 
> links, sorry - but they were real packaging-related issues). So I don't think 
> it makes sense to shorten the time window. I also don't think there's a need 
> to drop one version that's urgent - it's some effort and CI time, but the 
> balance is decent right now.

It's hard to say the balance is decent right now if the faster cadence
isn't even in full effect yet (as I noted in my original email).

Generally I would say that dropping support only means that users of
older versions would simply need to use an older version of the
library.

However this:

> We cannot do that (yet, at least). Failing to publish wheels for a supported 
> Python version on a major OS is far worse than dropping support completely. 
> This will remain true until the time that Pip starts defaulting to 
> wheels-only and never picks the sdist if there's an older release for that 
> platform + Python combo.

sounds like if even the latest version doesn't support a given CPython
version (and has CPython-versioned wheels), then pip installing it
will fail. Is that correct?

> There was a lot of discussion around how big the window should be with 42mo 
> being about in the middle of what people advocated for.  I am aware of 
> institutions that are on an every-other Python upgrade pattern (py37 -> py39 
> -> (expected) py311) so always having at least 3 Pythons in the window is 
> important.  Based on what we have seen so far, I still think 42mo is a good 
> choice, but do not think we have seen it in operations long enough to draw 
> any conclusions (the downside of year-scale planning is you need to wait 
> years to see if it worked out like you expected!) and hence do not think we 
> should make any changes to the time window for another few years.   That 
> said, I am currently sympathetic to making the window longer and against 
> making it shorter.

The reason I brought this up is because we were looking at whether it
makes sense to use NEP 29 for SymPy. Obviously we aren't bound to
using it, but given this apparent discrepancy in the text (which still
exists), I thought I would mention it here. But I will say that from
my point of view, supporting more Python versions is a burden. Having
more builds on CI means longer wait times to merge PRs. And having to
wait longer for new Python features is also annoying.

Aaron Meurer
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: Dropping the pdf documentation.

[Aaron Meurer]

If it's any help, I would suggest looking at how SymPy does its PDF
documentation. We have a few adjustments to the Sphinx defaults to make
things work (although it's honestly not that much, mainly just using XeTeX
for Unicode support).
https://github.com/sympy/sympy/blob/master/doc/Makefile

One of the things that was mentioned in the issue is SVG images. We found
in SymPy that the best way to convert SVG images to a format that can be
used for PDFs is to use Google Chrome (all the other SVG to PDF converters
are flawed in some way, see https://github.com/sympy/sympy/pull/22468 for
details). See this script
https://github.com/sympy/sympy/blob/master/doc/convert-svg-to-pdf.sh and
also this PR https://github.com/sympy/sympy/pull/23035. Perhaps the code in
that PR should be factored out into a separate Sphinx extension that can be
reused by other projects.

I personally don't use PDF documentation but I know for SymPy, many people
do, which is why we have continued to support it.

Also, one final thing I will note is that if you have any LaTeX math in
your documentation, the PDF build is the only way you can be sure that that
LaTeX is valid. MathJax only renders the LaTeX in the browser when the page
is loaded so you will only notice invalid math once you look at it and see
it gives an error. To contrast, the PDF docs fail at the build stage if the
LaTeX is invalid. Obviously we have a lot of math in the SymPy docs but I
don't know if there is any in the NumPy or SciPy docs. This does also mean
that to be useful you actually need to build it on CI. If you don't, you
will end up with docs that don't actually build any more by the release
because of unseen errors (even without LaTeX math this will likely end up
happening, so I would recommend against this).

Aaron Meurer

On Sun, May 22, 2022 at 2:08 PM Charles R Harris 
wrote:

> Hi All,
>
> This is a proposal to drop the generation of pdf documentation and only
> generate the html version. This is a one way change due to the difficulty
> maintaining/fixing the pdf versions. See minimal discussion here
> .
>
> Chuck
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: asmeu...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com

[Numpy-discussion] Re: NEP 29 and the faster Python release cadence

[Thomas Caswell]

I created https://github.com/numpy/numpy/pull/21601 to update NEP29 to
address PEP602.

I'm not sure what the procedure for updating the NEP is.  What I wrote may
be too editorial, we could amend it to "PEP602 changed cadence, we are not
reacting." with no explanation as well.

Tom

On Wed, May 25, 2022 at 10:41 PM Aaron Meurer  wrote:

> > I have seen problems popping up already in a few places with latest
> numpy not supported what is still the most commonly used Python version
> (don't have links, sorry - but they were real packaging-related issues). So
> I don't think it makes sense to shorten the time window. I also don't think
> there's a need to drop one version that's urgent - it's some effort and CI
> time, but the balance is decent right now.
>
> It's hard to say the balance is decent right now if the faster cadence
> isn't even in full effect yet (as I noted in my original email).
>
> Generally I would say that dropping support only means that users of
> older versions would simply need to use an older version of the
> library.
>
> However this:
>
> > We cannot do that (yet, at least). Failing to publish wheels for a
> supported Python version on a major OS is far worse than dropping support
> completely. This will remain true until the time that Pip starts defaulting
> to wheels-only and never picks the sdist if there's an older release for
> that platform + Python combo.
>
> sounds like if even the latest version doesn't support a given CPython
> version (and has CPython-versioned wheels), then pip installing it
> will fail. Is that correct?
>
> > There was a lot of discussion around how big the window should be with
> 42mo being about in the middle of what people advocated for.  I am aware of
> institutions that are on an every-other Python upgrade pattern (py37 ->
> py39 -> (expected) py311) so always having at least 3 Pythons in the window
> is important.  Based on what we have seen so far, I still think 42mo is a
> good choice, but do not think we have seen it in operations long enough to
> draw any conclusions (the downside of year-scale planning is you need to
> wait years to see if it worked out like you expected!) and hence do not
> think we should make any changes to the time window for another few years.
>  That said, I am currently sympathetic to making the window longer and
> against making it shorter.
>
> The reason I brought this up is because we were looking at whether it
> makes sense to use NEP 29 for SymPy. Obviously we aren't bound to
> using it, but given this apparent discrepancy in the text (which still
> exists), I thought I would mention it here. But I will say that from
> my point of view, supporting more Python versions is a burden. Having
> more builds on CI means longer wait times to merge PRs. And having to
> wait longer for new Python features is also annoying.
>
> Aaron Meurer
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: tcasw...@gmail.com
>


-- 
Thomas Caswell
tcasw...@gmail.com
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com