[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Nick Coghlan]

Change by Nick Coghlan :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
type:  -> compile error

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Petr Viktorin]

Petr Viktorin  added the comment:


New changeset d917cfe4051d45b2b755c726c096ecfcc4869ceb by Petr Viktorin (Jeroen 
Demeyer) in branch '3.8':
[3.8] bpo-37250: put back tp_print for backwards compatibility (GH-14193)
https://github.com/python/cpython/commit/d917cfe4051d45b2b755c726c096ecfcc4869ceb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Carol Willing]

Carol Willing  added the comment:

Thank you Petr. Looking at your response, the full discussion here, and the 
fact that we don't have an easy way to put a number on who would be impacted, I 
agree that merging the PR seems the best course of action.


Thanks Stefan for additional follow up with Cython too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Stefan Behnel]

Stefan Behnel  added the comment:

Thank you, Petr, for bringing this discussion back on a solid basis, actually 
twice already.

And sorry for causing this problem in the first place. The "tp_print = 0" was 
necessary to fix misbehaviour in Py2, and we should have restricted it to Py2 
back then (that line has been there since late 2013).

I think keeping some kind of a "tp_print" named field in PyTypeObject for now 
is the best we can do. I like Jeroen's PR (thanks for taking care of so many 
things at a time, Jeroen!), and I think we should just go with it.

I'm happy to release a new point release for Cython, as soon as it's clear what 
CPython will do, and how Cython can detect the change to adapt to it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Nick Coghlan]

Nick Coghlan  added the comment:

There's also the question of complying with our own deprecation policy, as the 
approval to ditch tp_print API compatibility in PEP 590 relied on the fact that 
it has been deprecated since Python 3.0.

However, it turns out the associated programmatic deprecation had never 
actually been implemented, which means it is now appropriate to provide that 
deprecation warning belatedly in Python 3.8, and then upgrade to full removal 
in Python 3.9.

That way the folks building affected sdists from source will get the additional 
warnings to tell them "Hey, you have a looming maintainability problem to solve 
(and a year or two to solve it)", rather than the current situation, which is 
"You have a dependency management crisis *now*, and until you solve it, you 
can't even start testing on Python 3.8, since you won't even be able to build 
your dependencies, let alone install them".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Petr Viktorin]

Petr Viktorin  added the comment:

1. If sdist consumers will need to regenerate an sdist?

They would need to compile from the source, e.g. clone from Git and make an 
sdist (or install) from that.

2. How many sdist consumers will likely be impacted?

I don't know. Depends on how fast people migrate to Python 3.8.

3. If impacted, how difficult would the regeneration be for a scientist who is 
moderately comfortable with compilation?

See (1.)

4. Is there any guidance or information that can be given to sdist consumers?

"Raise the issue with the sdist maintainers" is probably their best bet.

---

Realistically, I expect those people to stay on Python 3.7- until their 
dependencies are updated to (and also tested with!) 3.8.

I've seen anecdotal evidence of this hitting people who want to use real-world 
projects to benchmark Python 3.8+: 
https://mail.python.org/archives/list/python-...@python.org/message/TF2JI7G5ZMCGUMM3AWNSCQDYVFNRPMQ4/

If we "stop discussing and just [Apply PR 14193]", people wen't be blocked from 
trying out 3.8, even if sdist maintainers don't care about 3.8 yet.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Carol Willing]

Carol Willing  added the comment:

Thanks Petr for the wonderful summary.

Of the pros/cons that you mention (if I am understanding correctly), I think 
that the issue comes down to sdist consumers and regeneration.

1. If sdist consumers will need to regenerate an sdist?
2. How many sdist consumers will likely be impacted?
3. If impacted, how difficult would the regeneration be for a scientist who is 
moderately comfortable with compilation?
4. Is there any guidance or information that can be given to sdist consumers?

Do we have any good metrics/insights on Question 2 and 3?

---

While TypeObject size, updating a pinned version, and vectorcall optimization 
for prior versions are considerations, I believe that they are secondary:

- Typeobject size: Nice to have a smaller size but probably not a concern of 
the sdist consumer or general user.
- Update a pinned Cython version: General expectation is it should work with 
the environment which it was pinned within. There should not be an expectation 
that the pinned version will work if other parts of the environment change.
- vectorcall optimization extends to prior versions: While perhaps nice to 
have, it is likely more an added benefit instead of an implicit contract or 
expectation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> With this fixed, and the PyCode_New -> PyCode_New + PyCode_WithPosArgs fix 
> from bpo-37221 merged, then that long tail will get another few years of 
> life, and will start emitting deprecation warnings for the part that's 
> actually going to go away in Python 3.9 (i.e. the tp_print slot).

By definition (as unmaintained packages), that's a few more years until death. 
Nobody is looking at their compile time warnings (that would require 
maintenance ;) ), so nothing will happen until it actually breaks. Keeping the 
field for one more version makes less sense than keeping it forever.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> I'm curious what you mean here...

msg345409

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Nick Coghlan]

Nick Coghlan  added the comment:

It isn't the actively maintained projects that publish wheel files that I'm 
worried about fixing - it's the sdist only projects that would otherwise only 
need recompilation to work on the new Python version.

With this fixed, and the PyCode_New -> PyCode_New + PyCode_WithPosArgs fix from 
bpo-37221 merged, then that long tail will get another few years of life, and 
will start emitting deprecation warnings for the part that's actually going to 
go away in Python 3.9 (i.e. the tp_print slot).

My opinion would be different if tp_print had actually been emitting compile 
time warnings for the past 10 years, but it hasn't been - we just haven't been 
using it for anything.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> based on precedents that do not apply to this situation

I'm curious what you mean here...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

FWIW, I agree totally with Petr's analysis.

Jeroen's analysis ignores some points, has technical inaccuracies (nobody can 
use a static type compiled for 3.7 with 3.8), and is based on precedents that 
do not apply to this situation. (All of which I've mentioned above, but I 
assume nobody is reading the whole repetitive thread anymore.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Petr Viktorin]

Petr Viktorin  added the comment:

> scientific projects that depend on it may lag longer as rebuilds and releases 
> will need to happen on PyPI, conda, and conda-forge.

Keep in mind that binary artifacts (wheels, Conda/distro packages) should not 
be affected. They need to be re-built for Python 3.8 *either way*, because the 
ABI changed.

The only artifacts that putting tp_print back in helps are sdists that include 
pre-generated Cython output, plus possibly projects that pin Cython to an older 
version.

Specifically, for any platform without a C compiler installed, users aren't 
helped by putting tp_print back in.

So:

(1) Status-quo (keep everything as is in 3.8b1):
  - binary/compiled distributions need to be re-built (*1)
  - sdists need to be re-generated with an up-to-date Cython (*2)
  - projects that pin Cython need to update it.
  + pure source distributions (where compiling involves running Cython) works 
as before (*3)
  + type objects do not waste memory
  + Cython can implement vectorcall optimizations for past Python versions 
(before 3.8)

(2) Apply PR 14193:
  - binary/compiled distributions still need to be re-built (*1)
  + sdists DO NOT need to be re-generated (*2)
  + projects that pin Cython DO NOT need to update it.
  + pure source distributions (where compiling involves running Cython) works 
as before (*3)
  - type objects get 8 bytes larger
  + Cython can implement vectorcall optimizations for past Python versions 
(before 3.8)

(3) Put back tp_print the way it was in 3.7:
  - binary/compiled distributions still need to be re-built (*1)
  + sdists DO NOT need to be re-generated (*2)
  + projects that pin Cython DO NOT need to update it.
  + pure source distributions (where compiling involves running Cython) works 
as before (*3)
  - type objects get 8 bytes larger
  - Cython can NOT implement vectorcall optimizations for past (pre-3.8) Python 
versions

*1: applicable to most wheel consumers, conda users, etc.
*2: applicable to sdist consumers with a compiler
*3: should be applicable to Conda/Fedora/Debian/... packagers (not so much end 
users)

> Is four months really an unreasonable timeline? I thought most of these 
> packages released more frequently than CPython.

Note that maintainers of these packages need to take an action (build wheels) 
anyway to support Python 3.8 (on compiler-less platforms). They need to test, 
and ideally even read "Porting to Python 3.8" and adjust the code.
Is re-generating Cython output, and bumping the version, too much of an *added* 
burden here?

If a project doesn't release 3.8 wheels by Python 3.8.0rc1, we don't get 
feedback on it from PyPI users (but Conda & Linux distros should be unaffected, 
if they run Cython on build).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> Is four months really an unreasonable timeline? I thought most of these 
> packages released more frequently than CPython.

I'm mostly worried about dependency chains. Right now, any project which 
depends on numpy cannot even test on 3.8b1 because numpy doesn't build.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> Perhaps it would be helpful for someone to recap the current options on the 
> table for moving forward and the most significant risk to Cython, Projects 
> using Cython, or CPython by selecting each option.

There are three options:

(1) Status-quo (keep everything as is in 3.8b1): this breaks any Cython project 
which has not been recythonized with the latest Cython. For example, it is 
(together with bpo-37221 which is planned to be fixed) the reason that "pip 
install numpy" doesn't work with 3.8b1. Typically, projects run Cython at 
packaging time, so this needs a new source release of every single Cython-using 
package.

(2) Apply PR 14193: the only disadvantage that I can think of is that types 
would get 8 bytes larger. We should stop discussing and just do this.

(3) Put back tp_print the way it was in 3.7: same as (2), except that it goes 
against PEP 590 as putting tp_vectorcall_offset in place of tp_print was a 
design decision made in that PEP. It would break ABI (any static type compiled 
with 3.7 would break on 3.8) and it prevents Cython from backporting vectorcall 
to 3.7. It's also a larger change compared to 3.8b1, so it's more work and more 
risky. In my personal opinion, the worst of the three options.

Then there is the further question whether we should apply this only on 3.8 or 
possibly also later versions. But here the consensus seems to be 3.8 only.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Carol Willing]

Carol Willing  added the comment:

> Is four months really an unreasonable timeline? I thought most of these 
> packages released more frequently than CPython.

I can't really predict with any accuracy what the release cycle is for most 
scientific projects. Anywhere from 1 month to 9 months probably covers 90% of 
the actively maintained scientific projects. So is four months unreasonable? 
Not entirely unreasonable IMHO. Six months seems a safer estimate. What makes 
it difficult to predict is that many of the scientific projects do not have 
rigid release cycles and push out releases when a specific feature or bug fix 
warrants it.

Perhaps it would be helpful for someone to recap the current options on the 
table for moving forward and the most significant risk to Cython, Projects 
using Cython, or CPython by selecting each option.

Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> While Cython may be able to respond within the beta period, scientific 
> projects that depend on it may lag longer as rebuilds and releases will need 
> to happen on PyPI, conda, and conda-forge. Release versioning is critical in 
> the science world as we depend on it for scientific research reproducibility. 
> While we may not come up with an ideal solution for everyone, let's try to 
> find something acceptable for Cython and the Core Python devs/users.

Cython has already responded and made a new release, so the problem would be 
whether the scientific projects will release new sdists some time in the next 
four months (they will almost all have to do new wheels for 3.8 too).

Is four months really an unreasonable timeline? I thought most of these 
packages released more frequently than CPython.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Carol Willing]

Carol Willing  added the comment:

Petr, Thanks for the thoughtful summary.

If I am understanding the many messages in this and the other related issue and 
by looking at this with a scientific Python hat on (not as the Steering 
Council), I'm inclined to favor Nick's approach if possible:

> 1. In 3.8.0b2, move tp_print to the end of the structure (rather than 
> removing it entirely) to make the "type->tp_print" in the old Cython 
> generated code harmless rather than a compilation error
>2. In 3.9.0a1, remove it entirely again

I agree that we shouldn't punish our users. IMHO Cython should get the benefit 
of a compromise, though not a long term guarantee, related to tp_print since 
Cython addresses a very real need for performance in scientific code. 

While Cython may be able to respond within the beta period, scientific projects 
that depend on it may lag longer as rebuilds and releases will need to happen 
on PyPI, conda, and conda-forge. Release versioning is critical in the science 
world as we depend on it for scientific research reproducibility. While we may 
not come up with an ideal solution for everyone, let's try to find something 
acceptable for Cython and the Core Python devs/users.

--
nosy: +willingc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

If we're in doubt, we should just apply PR 14193. It's essentially a one-line 
patch that fixes actual problems experienced by users and which doesn't break 
anything. Why should we *not* apply it?

I know that we're not obliged to fix anything, but that is besides the point 
given that a fix exists.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> Note that the PR for the other similar issue #37221 has been accepted for 
> merging.

I've already explained why that issue is different (it breaks a compatibility 
guarantee; this one changes something that was explicitly deprecated).

Petr's question seems to be the right one that we have not yet answered:

> Is (the rest of) the beta cycle enough for enough of the ecosystem to release 
> new wheels, or should we give them the whole 3.9 cycle?

(And since the 3.8 RM has not been here to discuss release cycle stuff, let's 
invite him in)

--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

Can we then bring back the discussion to the core of this issue: should we 
restore tp_print or not in PyTypeObject?

Note that the PR for the other similar issue #37221 has been accepted for 
merging.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> I was trying to tell you that already in msg345969 but you insisted in 
> msg345988 that we should support that anyway.

I made a general note that "extremely unlikely" isn't unlikely enough to be an 
argument.

You'll notice I didn't quote any of the rest of your specific scenario, because 
I wasn't trying to address it. And in any case, you left out the cross-version 
part in that description, so you did actually describe a supported case.

(I'm not going to argue this particular point any further. If I can't help you 
learn the ways we do decision making on a project like CPython then I'll let 
someone else try.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Petr Viktorin]

Petr Viktorin  added the comment:

I agree with Steve that renaming tp_print to tp_vectorcall_offset is well 
within our rights.

For reference, the rationale for the replacing is the first section of:
   https://mail.python.org/pipermail/python-dev/2018-June/153927.html 
I hope Cython's situation here has not changed. (In that case, we should bring 
tp_print back, and add tp_vectorcall_offset to the end.)


Now, I see the question of bringing tp_print back as a *practical* 
compatibility matter, a workaround for the combination of two unfortunate 
issues:
- Cython sets tp_print in Python 3 (which is a bug, but honestly, the word 
"reserved" could very well be interpreted as "unused" rather than "don't touch 
this" -- and even the post linked above, and by extension PEP 590, implies that 
interpretation!)
- Packages hard-code Cython-generated files (which is a workaround for a 
long-standing setuptools limitation that is only recently being addressed).

If we bring tp_print back, it would be because *we shouldn't punish users* even 
if we can. Nobody did anything wrong, apart from a quite understandable 
bug/misunterstanding.


Note that 3rd party projects *do not* have the whole beta cycle to adapt. Until 
NumPy works, its dependents can't start adapting. (Unless adapting is done 
without PyPI packages, like the Fedora rebuild where we now "re-Cythonize" 
everything.)
On the other hand, 3rd party projects don't really *need* to support 3.8.0 as 
soon as it's released. It's just that if they don't support the betas and RCs, 
we're don't get feedback for them in time.


So I think the question is: Is (the rest of) the beta cycle enough for enough 
of the ecosystem to release new wheels, or should we give them the whole 3.9 
cycle?

I believe 3.8 beta2 is enough, and we don't need to bring tp_print back.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

The fact that tp_vectorcall_offset replaces tp_print was a very conscious 
decision that we made already when designing the PEPs (both PEP 580 and PEP 
590). We shouldn't just throw that away without a good reason.

So far I haven't seen any good reason, only an unsubstantiated claim that it is 
supposedly more backwards compatible to put tp_vectorcall_offset at the end.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> This isn't a supported scenario though.

Of course it's not! I was trying to tell you that already in msg345969 but you 
insisted in msg345988 that we should support that anyway.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> a class compiled with 3.7 could subclass a class compiled with 3.8 which uses 
> vectorcall.

This isn't a supported scenario though. The only way to mix extension modules 
compiled against different Python versions is to use the limited ABI, which 
does not include PyTypeObject. And PyType_FromSpec is going to allocate a 
PyTypeObject according to the version of Python that is running, not the one it 
was compiled against. So even if someone could force this scenario, it's not 
one we have to worry about.

> The idea is just to give more time (one Python major release) to maintainers 
> of C extensions to regenerate their Cython .c files.

If the beta cycle isn't long enough for this, then we need a longer beta cycle, 
since that's the whole point of it. Projects that can't make at least one new 
release with update Cython in time for Python 3.8 are presumably completely 
unmaintained - it's not like the 3.8 release is a huge surprise - and if 
completely unmaintained they aren't going to get a new release for 3.9 either. 
So either we fix them permanently by not removing tp_print ever, or we give 
them an entire beta release period to update their code for the new version of 
Python.

There's a discussion at 
https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828
 about updating the release cycle, and a couple of PEPs being written. That 
would be a good place to suggest that our current beta release is not long 
enough for packages to adapt to new releases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[STINNER Victor]

STINNER Victor  added the comment:

> If we should put in the workaround for 3.8, how does that make it okay to 
> remove in 3.9?

Cython has been modified to no longer set tp_print. The idea is just to give 
more time (one Python major release) to maintainers of C extensions to 
regenerate their Cython .c files.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> We haven't released vectorcall yet, so it has no compatibility baseline.

I'm talking about the "extremely unlikely" scenario mentioned earlier: a class 
compiled with 3.7 could subclass a class compiled with 3.8 which uses 
vectorcall.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> It will break ABI compatibility in any case where vectorcall is used, while 
> my proposal of putting tp_print at the end does not.

ABI compatibility with what? We haven't released vectorcall yet, so it has no 
compatibility baseline.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> If compatibility is the concern here, then we should be aiming for the 
> smallest diff between 3.7 and 3.8

I challenge that assertion.

The smallest diff may superficially look like the best solution, but on closer 
inspection, it is not. It will break ABI compatibility in any case where 
vectorcall is used, while my proposal of putting tp_print at the end does not.

What you propose is *worse* than the status quo.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> Can you give a least an argument of why that's better than putting back 
> tp_print at the end?

Minimal change, that's why. If there are compatibility issues with the current 
change that we need to fix, then we should make the least incompatible change 
possible, which is adding the new field at the end of the struct.

> I gave 3 why that's NOT a good idea: msg345357

Those are equally good reasons for putting tp_print back where it was, so I'm 
not sure what your point is?

Compatibility has to be measured against the 3.7 release, not previous 3.8 
prereleases. If compatibility is the concern here, then we should be aiming for 
the smallest diff between 3.7 and 3.8, even if that means we change things 
significantly between 3.8.0b1 and b2.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> If we should put in the workaround for 3.8, how does that make it okay to 
> remove in 3.9?

I can easily change my PR to keep it in 3.9, no problem. Nick, what do you 
think?

> put the new field at the end and leave the deprecated one exactly as it was.

Can you give a least an argument of why that's better than putting back 
tp_print at the end? I gave 3 why that's NOT a good idea: msg345357

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

If we should put in the workaround for 3.8, how does that make it okay to 
remove in 3.9? It's been deprecated since 3.0 already, so if we can't remove it 
after eight releases, I don't see how it's consistent to remove it after a 
ninth. Once we put it back, it basically has to stay.

The middle ground doesn't make sense. We should either replace it with the new 
field, or put the new field at the end and leave the deprecated one exactly as 
it was.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Nick Coghlan]

Nick Coghlan  added the comment:

(Note: this isn't SC feedback yet, it's individual contributor feedback from me)

As others noted, the reason we treat "Incompatible with Cython generated source 
code" distinctly from other C API source compatibility issues is because 
Cython-using projects often add the generated files to their source tarballs, 
so install targets only need a C compiler, not Cython itself. Even when it's a 
case where the older Cython generated code wasn't doing the right thing, 
there's pragmatic value in smoothing those transitions.

That said, I still agree with PEP 590's decision to re-use the existing reserve 
function pointer slot, rather than adding a new one to the end of the type 
structure indefinitely, which would make the compatibility measure:

1. In 3.8.0b2, move tp_print to the end of the structure (rather than removing 
it entirely) to make the "type->tp_print" in the old Cython generated code 
harmless rather than a compilation error
2. In 3.9.0a1, remove it entirely again

That way we still end up in the place we were aiming to get in PEP 590, we just 
take an extra step in getting there.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[STINNER Victor]

STINNER Victor  added the comment:

Steve Dower:
> * Pre-generated Cython .c files can be easily regenerated, and given their 
> use of low-level and internal APIs would cause undue compatibility burden on 
> CPython if we were to treat everything it uses as public stable API

It seems like you missed the whole point of this issue. No, it's not "easy" to 
regenerate these Cython .c files.

Let's pick a random module on PyPI... numpy.

$ python3.8 -m venv env
$ env/bin/python -m pip install numpy
...
Collecting numpy
  Downloading 
https://files.pythonhosted.org/packages/d3/4b/f9f4b96c0b1ba43d28a5bdc4b64f0b9d3fbcf31313a51bc766942866a7c7/numpy-1.16.4.zip
 (5.1MB)
...
numpy/random/mtrand/mtrand.c:311:11: erreur: trop peu d'arguments pour la 
fonction « PyCode_New »
  311 |   PyCode_New(a, k, l, s, f, code, c, n, v, fv, cell, fn, 
name, fline, lnos)
  |   ^~


So right now, 16 days after Cython 0.29.10 release, numpy cannot be installed 
on Python 3.8 because of bpo-37221. But bpo-37221 is going to revert 
PyCode_New() API, and then the install will fail at "tp_print = NULL".

numpy is like one of the most popular C extensions on PyPI. But there are 184k 
projects on PyPI: some of them of Cython.

Some old projects on PyPI will never be updated anymore, but there are likely 
applications in applications which rely on them. This problem is very common.

Random example: setuptools is another top module on PyPI and it vendors 
html5lib. But html5lib didn't get a release for 1 year 1/2. html5lib emits a 
deprecation warning which now blocks bpo-37324 ("collections: remove deprecated 
aliases to ABC classes").

--

This issue is a practical problem: tons of projects on PyPI are currently 
broken on Python 3.8 and it will take *years* to update all of them.

IMHO we should reintroduce tp_print just to not break all these projects. 8 
bytes per PyTypeObject is nothing compared to annoyance of issues introduced by 
tp_print removal.

--

Yes, I know that tp_print was deprecated for 10 years and it never emitted any 
deprecation warning. And people don't pay attention to warnings until it 
becomes a hard error...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> Personally, I don't think that your "read of it" in #msg345990 is relevant to 
> the discussion of the PR (I don't mean this in a bad way).

It's relevant to the *existence* of the PR, because when you follow through the 
rest of the thought process, there is no need for a PR at all, and so no need 
to discuss it.

There's no reason to assume that a PR is required just because an issue was 
opened.

If we'd broken actual functionality in CPython or Cython, then there'd be a bug 
to consider. But all we broke is a misuse of a reserved field, and if it were 
in a single project then we wouldn't have spent anywhere near this amount of 
time discussing it - it's *only* because Cython was putting it in all its 
generated code that there's any reason to even look at this, as it could have a 
disproportionate effect if we believe it's a fair assumption that 
Cython-generated code targeting 3.7 should also work on 3.8. I've stated my 
opinion on that, but a decision either requires consensus from those of us 
involved here (Victor and Stefan) or an appeal to a higher authority (steering 
council).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

Personally, I don't think that your "read of it" in #msg345990 is relevant to 
the discussion of the PR (I don't mean this in a bad way).

It's one thing to say "we're not obliged to fix this" (I agree with that) but 
that doesn't imply that it's forbidden to fix the issue and that we should 
automatically reject proposed fixes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

I would argue for the pragmatic solution: PR 14193 fixes that old 
Cython-generated code. More generally, it fixes API compatibility and doesn't 
make ABI compatibility worse than the status quo (the proposal of putting back 
tp_print in place of tp_vectorcall_offset does).

Of course, it may not fix every single use case, but even if we can't fix that 
hypothetical 0.1%, we can still fix the 99.9%.

The proposed fix is very small and only applied to 3.8, so there is no 
long-term maintenance burden. So the obvious question: why not? What's wrong 
with PR 14193?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> Don't confuse *ABI* and *API* compatibility. For *API* compatibility, it 
> doesn't matter where we put tp_print.

Don't worry, I'm not. (Though I did blur the lines for the sake of a 
tongue-in-cheek reply to your comment. Probably shouldn't do that in text with 
people I haven't spent time with in real life :) )

We're not making any progress here, so perhaps it's time to escalate the 
decision to the steering council. My read of it:

* Python 3.8 removed a deprecated struct member
* Cython prior to 0.29.10 was writing directly to this member to clear it
* any sdists that include pregenerated Cython modules will fail to build 
against 3.8

Things to decide (beyond this one-off case):
* are deprecated struct members allowed to be removed in new major version?
* should Cython (using non-stable API) be expected to make updates for new 
major CPython versions?
* should we treat pre-generated Cython .c files with the same compatibility 
constraints we use for hand-written C code

My view is yes, yes and no.
* If a field has been deprecated for the normal amount of time and is not in 
the stable API, it can be removed.
* Cython does not use the stable API, and so should expect to make changes when 
a new major version of CPython is released
* Pre-generated Cython .c files can be easily regenerated, and given their use 
of low-level and internal APIs would cause undue compatibility burden on 
CPython if we were to treat everything it uses as public stable API

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> Oh? In that case, let's just remove the reserved/deprecated field :)

Don't confuse *ABI* and *API* compatibility. For *API* compatibility, it 
doesn't matter where we put tp_print.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Steve Dower]

Steve Dower  added the comment:

> If we reintroduce it, why not put it back at its previous place, to provide 
> ABI compatibility?

I agree. If compatibility matters here, then we should add new elements at the 
end.

> First of all, we longer care about ABI compatibility of PyTypeObject.

Oh? In that case, let's just remove the reserved/deprecated field :)

> Second, it is extremely unlikely ...

As a rule, "extremely unlikely" isn't unlikely enough for a product with as 
much reach as Python has. Even if something only happens to 0.1% of users, 
that's still thousands of people. You have to be prepared to justify hurting 
those people - potentially to their faces - to back up ideas like this. (And 
FWIW, yes, I'm happy to explain to people that their pre-generated Cython code 
broke because Cython had a bug that has since been fixed.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Jeroen Demeyer  added the comment:

> If we reintroduce it, why not put it back at its previous place, to provide 
> ABI compatibility?

First of all, we longer care about ABI compatibility of PyTypeObject.

Second, it is extremely unlikely that the class will use vectorcall: the only 
way would be to inherit from a base class that uses vectorcall, but there 
aren't any subclassable classes using vectorcall in CPython. If the class 
doesn't use vectorcall, then it really doesn't matter: tp_vectorcall_offset is 
unused in that case, just like tp_print.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[STINNER Victor]

STINNER Victor  added the comment:

> I just realized that there is a much simpler solution: add back tp_print at 
> the *end* of the PyTypeObject structure. This will fix the Cython problem and 
> is even less likely to break stuff. I'm doing that in PR 14193.

If we reintroduce it, why not put it back at its previous place, to provide ABI 
compatibility?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37250] C files generated by Cython set tp_print to 0: PyTypeObject.tp_print removed

[Jeroen Demeyer]

Change by Jeroen Demeyer :


--
title: C files generated by Cython set tp_print to NULL: PyTypeObject.tp_print 
removed, replaced with tp_vectorcall_offset -> C files generated by Cython set 
tp_print to 0: PyTypeObject.tp_print removed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com