date:20220201

It seems to me that a big part of the problem is that Cython feels entitled
to use arbitrary CPython internals. Another part is that there doesn't seem
to be any Cython maintainer interested in communicating with the core devs
who are changing those CPython internals. We have to resort to creating
issues in the Cython tracker and wait weeks before someone responds, and
often not in a supportive way.

I am fully aware that Cython is an important tool in our ecosystem. But
before I agree that we should roll back things "because it breaks Cython" I
would like to see a lot more participation in CPython's development by
Cython developers. (Who are they even? I only know "scoder" -- who else can
speak authoritatively on behalf of Cython?)

During a beta cycle I would see the roles reversed. But until late May we
are still working on alphas. If Cython wants to wait until beta 1 that's
fine, but then their input on the design of CPython changes is necessarily
much more limited.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RBL3K65CULUNYKMSSIPEWUDCN4CSUA5Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

Christian Heimes schrieb am 01.02.22 um 16:42:

On 01/02/2022 16.08, Victor Stinner wrote:

I would prefer to introduce C API incompatible changes differently:
first fix Cython, and *then* introduce the change.

Note: Fedora doesn't need (3) since we always regenerated Cython code in
numpy.

this is a reasonable request for beta releases, but IMHO it is not feasible
for alphas. During alphas we want to innovate fast and play around. Your
proposal would slow down innovation and impose additional burden on core
developers.

Let's at least try not to run into a catch-22.

I'm reluctant to working on adapting Cython during alphas, because it
happened more than once that incompatible changes in CPython were rolled
back or modified again during alpha, beta and rc phases. That means more
work for me and the Cython project, and its users. Code that Cython users
generate and release on their side with a release version of Cython will
then be broken, and sometimes even more broken than with an older Cython
release.

But Victor is right, OTOH, that the longer we wait with adapting Cython,
the longer users have to wait with testing their code in upcoming CPython
versions, and the higher the chance of post-beta and post-rc rollbacks and
changes in CPython.

I don't have the capacity to follow all relevant changes in CPython,
incompatible or not. Even a Cython CI breakage of the CPython-dev job
doesn't always mean that there is something to do on our side and is
therefore silenced to avoid breakage of our own project workflows, and to
be looked at irregularly. Additionally, since Cython is a crucial part of
the Python ecosystem, breakage of Cython by CPython sometimes stalls the
build pipelines of CI images, which means that new CPython dev versions
don't reach the CI servers for a while, during which the breakage will go
even more unnoticed.

I think you should generally appreciate Cython (and the few other C-API
abstraction tools) as an opportunity to get a large number of extensions
adapted to CPython's now faster development all at once. The quicker these
tools adapt, the quicker you can get user feedback on your own changes, and
the more time you have to validate and refine them during the alpha and
beta cycles.

You can even see the adaptation as a way to validate your own changes in
the real world. It's cool to write new code, but difficult to find out
whether it behaves the way you want for the intended audience. So – be part
of your own audience.

Stefan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/LJDI74V4IOHPCMQUEGH6VIQWHLM3MADG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

2022-02-01 Thread Irit Katriel via Python-Dev

Stefan,

There two separate issues here. One is the timing of committing changes into
cython, and the other is the process by which the cython devs learn about
cpython development.

On the first issue, you wrote:

I'm reluctant to working on adapting Cython during alphas, because it
> happened more than once that incompatible changes in CPython were rolled
> back or modified again during alpha, beta and rc phases. That means more
> work for me and the Cython project, and its users. Code that Cython users
> generate and release on their side with a release version of Cython will
> then be broken, and sometimes even more broken than with an older Cython
> release.
>

I saw in your patch that you make changes such that they impact only the
new cpython version. So for old versions the generated code should not be
broken. Surely you don't guarantee that cython code generated for an alpha
version of cpython will work on later versions as well?  Users who generate
code for an alpha version should regenerate it for the next alpha and for
beta, right?

On the second issue:

I don't have the capacity to follow all relevant changes in CPython,
> incompatible or not.

We get that, and this is why we're asking to work with you on cython updates
so that this will be easier for all of us. There are a number of cpython
core devs
who would like to help cython maintenance. We realise how important and
thinly resourced cython is, and we want to reduce your maintenance burden.
With better communication we could find ways to do that.

Returning to the issue that started this thread - how do you suggest we
proceed with the exc_info change?

Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MBGJFYWKBFBUKNA2MXJ5THX5M657J5WH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python


On 2/02/22 5:42 am, Victor Stinner wrote:

There is an on-going effort adding getter and setter functions on two
structures which are causing most troubles on Python updates:

* PyThreadState: https://bugs.python.org/issue39947
* PyFrameObject: https://bugs.python.org/issue40421


In the case of PyFrameObject, as far as I know the only reason
Cython needs to mess with it is to get filename/lineno information
into tracebacks. If that's still true, I think it would be better
to make it possible to add that information directly to traceback
objects so that fake frame objects are not needed.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VNNASOMXNFHRJYCUZLPWRWMXVNUIKEOU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python


On 2/02/22 8:48 am, Guido van Rossum wrote:
It seems to me that a big part of the problem is that Cython feels 
entitled to use arbitrary CPython internals.


I think the reason for this is that Cython is trying to be two
things at once: (1) an interface between Python and C, (2) a
compiler that turns Python code into fast C code.

To address this there could be an option to choose between
"compatible code" and "fast code", with the former restricting
itself to the stable API.

--
Greg

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EACB7ZZVDDNL4QAIODYDNWLKI455QDKP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

2022-02-01 Thread Christopher Barker

On Tue, Feb 1, 2022 at 2:36 PM Greg Ewing 
wrote:

> I think the reason for this is that Cython is trying to be two
> things at once: (1) an interface between Python and C, (2) a
> compiler that turns Python code into fast C code.
>

As a long time Cython user, but not a Cython developer, I think (2) is the
primary purpose, with (1) as a handy side benefit (otherwise we'd just use
ctypes, yes?)

That being said a not-quite-as-fast-as-possible mode would be fine.

Though I'm not sure it would buy much, as long as projects (including major
ones like numpy) are using as-fast-as-possible mode.

As long as I'm being wishy-washy: maybe we don't need as-fast-as-possible
at all, and can get to fast-enough-that-you-won't-notice. e.g. if the
stable API is missing a feature needed for important performance reasons,
then it could be extended, rather than forcing projects that use it to
suffer significantly performance-wise.

It seems the Cython/numpy use-case is a good way to test the limits of
performance.

-CHB

> To address this there could be an option to choose between
> "compatible code" and "fast code", with the former restricting
> itself to the stable API.
>
> --
> Greg
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EACB7ZZVDDNL4QAIODYDNWLKI455QDKP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BV5JDJE66YGNTNJJ2R3IANBKB2M4LBBY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

2022-02-01 Thread dw-git

Greg Ewing wrote:
> To address this there could be an option to choose between
> "compatible code" and "fast code", with the former restricting
> itself to the stable API.

To some extent, that exists at the moment - many of the real abuses of the 
CPython internals can be controlled by setting C defines. For the particular 
feature that caused this discussion the majority of the uses can be turned off 
by defining CYTHON_USE_EXC_INFO_STACK=0 and CYTHON_FAST_THREAD_STATE=0. 
(There's still a few uses relating to coroutines, but those too flags are 
sufficient to get Cython to build itself and Numpy on Python 3.11a4).

Obviously it could still be better. But the desire to support PyPy (and the 
beginnings of the limited API) mean that Cython does actually have alternate 
"clean" code-paths for a lot of cases.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q3IQUKU35GNCUXBCK55JZ3B42LSVS2M2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

Hm... So maybe the issue is either with Cython's default settings (perhaps
traditionally it defaults to "as fast as possible but relies on internal
APIs a lot"?) or with the Cython settings selected by default by projects
*using* Cython?

I wonder if a solution during CPython's rocky alpha release cycle could be
to default (either in Cython or in projects using it) to the "not quite as
fast but not relying on a lot of internal APIs" mode, and to switch to
Cython's faster mode only once (a) beta is entered and (b) Cython has been
fixed to work with that beta?

Sure, occasionally things still change during beta, but the point of beta
is that things shouldn't change unless it is to fix bugs. On behalf of the
Faster CPython project I can commit to that for our contributions, we'll do
our advanced work on the 3.12 branch once 3.11beta has started.

All this is assuming that Cython's default can be adjusted independently
for CPython's upcoming release (3.11, for now) and separately for previous
releases (3.10 and before). But if it can't yet, surely *that* would be a
relatively simple change?

On Tue, Feb 1, 2022 at 3:07 PM  wrote:

> Greg Ewing wrote:
> > To address this there could be an option to choose between
> > "compatible code" and "fast code", with the former restricting
> > itself to the stable API.
>
> To some extent, that exists at the moment - many of the real abuses of the
> CPython internals can be controlled by setting C defines. For the
> particular feature that caused this discussion the majority of the uses can
> be turned off by defining CYTHON_USE_EXC_INFO_STACK=0 and
> CYTHON_FAST_THREAD_STATE=0. (There's still a few uses relating to
> coroutines, but those too flags are sufficient to get Cython to build
> itself and Numpy on Python 3.11a4).
>
> Obviously it could still be better. But the desire to support PyPy (and
> the beginnings of the limited API) mean that Cython does actually have
> alternate "clean" code-paths for a lot of cases.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Q3IQUKU35GNCUXBCK55JZ3B42LSVS2M2/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FUA5JDEN6374FDOZQWPZYHXH5E3U4V24/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python


On 2/02/22 11:53 am, Christopher Barker wrote:
As a long time Cython user, but not a Cython developer, I think (2) is 
the primary purpose, with (1) as a handy side benefit (otherwise 
we'd just use ctypes, yes?)


Personally, no, I would not "just use ctypes". The main reason I
created Pyrex was to avoid the extreme amounts of pain involved
in doing things like that.


That being said a not-quite-as-fast-as-possible mode would be fine.

Though I'm not sure it would buy much, as long as projects (including 
major ones like numpy) are using as-fast-as-possible mode.


That's why I think compatible mode should be the default. Then
those who choose otherwise will be aware of what they are doing.

It would also mean that CPython developers needn't have as many
qualms about changing internals. Cython users would be in the same
position as someone writing an extension module by hand and
choosing whether to stick to the stable API or not. If they rely
on internals, they do so at their own risk.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WTDYDJYNS7XXMXI7LVN5GOE4CVAKUIZ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

2022-02-01 Thread Christopher Barker

On Tue, Feb 1, 2022 at 3:22 PM Greg Ewing 
wrote:

> On 2/02/22 11:53 am, Christopher Barker wrote:
> > As a long time Cython user, but not a Cython developer, I think (2) is
> > the primary purpose, with (1) as a handy side benefit (otherwise
> > we'd just use ctypes, yes?)
>
> Personally, no, I would not "just use ctypes". The main reason I
> created Pyrex was to avoid the extreme amounts of pain involved
> in doing things like that.
>

And thanks for that! I too find ctypes painful. But the other reason I use
Cython (and Pyrex before it) even when I need to wrap C code, is that I can
make a "thick" high performance wrapper, e.g. if I want to call an
expensive C function on each item in a sequence, I can do that in Cython,
removing a lot of the overhead of Python.

Anyway, I don't think there's any disagreement that high-performing Cython
code is an important use case.

> That being said a not-quite-as-fast-as-possible mode would be fine.
>
> > Though I'm not sure it would buy much, as long as projects (including
> > major ones like numpy) are using as-fast-as-possible mode.
>
> That's why I think compatible mode should be the default. Then
> those who choose otherwise will be aware of what they are doing.
>

Sure, but this thread is not just about users like me, that can choose the
more stable way or the faster way, but specifically about numpy, which is
going to use the fast way -- and we don't want to break that any more than
absolutely necessary.

-CHB

-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PK2MRMIRDDWMKLFC5MIAPBRVGLHNYQ2E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

Greg Ewing schrieb am 01.02.22 um 23:33:

On 2/02/22 8:48 am, Guido van Rossum wrote:
It seems to me that a big part of the problem is that Cython feels
entitled to use arbitrary CPython internals.

I think the reason for this is that Cython is trying to be two
things at once: (1) an interface between Python and C, (2) a
compiler that turns Python code into fast C code.

To address this there could be an option to choose between
"compatible code" and "fast code", with the former restricting
itself to the stable API.

There is even more than such an option. We use a relatively large set of
feature flags that allow us to turn the usage of certain implementation
details of the C-API on and off. We use this to adapt to different Python
C-API implementations (currently CPython, PyPy, GraalPython and the Limited
C-API), although with different levels of support and reliability.

Here's the complete list of feature sets for the different targets:

https://github.com/cython/cython/blob/5a76c404c803601b6941525cb8ec8096ddb10356/Cython/Utility/ModuleSetupCode.c#L56-L311

This can also be used to enable and disable certain dependencies on CPython
implementation details, e.g. PyList, PyLong or PyUnicode, but also type
specs versus PyTypeObject structs.

Most of these feature flags can be disabled by users. There is no hard
guarantee that this always works, because it's impossible to test all
combinations, and then there are bugs as well, but most of the flags are
independent, which should usually allow to disable them independently.

So, one of the tools that we have in our sleeves when it comes to
supporting new CPython versions is also to selectively disable the
dependency on a certain C-API feature that changed, at least until we have
a way to adapt to the change itself.

In the specific case of the "exc_info" changes, however, that didn't quite
work, because that change was really not anticipated at that level of
impact. But there is an implementation for Cython 3.0 alpha now, and we'll
eventually have a legacy 0.29.x release out that will also adapt in one way
or another. Just takes a bit more time.

Stefan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/QPAWLCS2FINPLVSDFFQCMVIELXETKQ3W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

2022-02-01 Thread Thomas Caswell

I disagree with point (3).

I think it would be better to discourage projects from including the output
of cython in their sdists.  They should either have cython as a build-time
requirement or provide built wheels (which are specific a platform and
CPython version).  The middle ground of not expecting the user to have
cython while expecting them to have a working c-complier is a very narrow
case and I think asking those users to install cython is worth the forward
compatibility for Python versions you get by requiring people installing
from source to re-cythonize.

Tom

-- 
Thomas Caswell
tcasw...@gmail.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LJX5X7NH6F6TD2XN2VV5PSEFR52R2MU7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

Hi Irit,

Irit Katriel via Python-Dev schrieb am 01.02.22 um 23:04:

There two separate issues here. One is the timing of committing changes into
cython, and the other is the process by which the cython devs learn about
cpython development.

On the first issue, you wrote:

I'm reluctant to working on adapting Cython during alphas, because it

happened more than once that incompatible changes in CPython were rolled
back or modified again during alpha, beta and rc phases. That means more
work for me and the Cython project, and its users. Code that Cython users
generate and release on their side with a release version of Cython will
then be broken, and sometimes even more broken than with an older Cython
release.

I saw in your patch that you make changes such that they impact only the
new cpython version. So for old versions the generated code should not be
broken. Surely you don't guarantee that cython code generated for an alpha
version of cpython will work on later versions as well? Users who generate
code for an alpha version should regenerate it for the next alpha and for
beta, right?

I'd just like to note that we are talking about three different projects
and dependency levels here (CPython, Cython and a project that uses
Cython), all three have different release cycles, and not all projects can
afford to go through a new release with a new Cython version regularly or
on the "emergency" event of a new CPython release. Some even don't provide
wheels and require their users to do a source build on their side. Often
with a fixed Cython version dependency, or even with pre-generated and
shipped C sources, which makes it harder for the end users to upgrade
Cython as a work-around.

But at least it should be as easy for the maintainers as updating their
Cython version and pushing a new release. In most cases. And things are
also becoming easier these days with improvements in the packaging
ecosystem. It can just take a bit until everyone has had the chance to
upgrade along the food chain.

On the second issue:

I don't have the capacity to follow all relevant changes in CPython,
incompatible or not.

We get that, and this is why we're asking to work with you on cython updates
so that this will be easier for all of us. There are a number of cpython
core devs
who would like to help cython maintenance. We realise how important and
thinly resourced cython is, and we want to reduce your maintenance burden.
With better communication we could find ways to do that.

I'm sure we will. Thanks for your help. It is warmly appreciated.

Returning to the issue that started this thread - how do you suggest we
proceed with the exc_info change?

I'm not done sorting out the options yet. Regarding CPython, I think it's
best to keep the current changes in there. It should be easier for us to
continue from where we are now than to adapt again to a revert in CPython.

Stefan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/BHIQL4P6F7OPMCAP6U24XEZUPQKI62UT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

Guido van Rossum schrieb am 02.02.22 um 00:21:

On Tue, Feb 1, 2022 at 3:07 David wrote:

Greg Ewing wrote:

To address this there could be an option to choose between
"compatible code" and "fast code", with the former restricting
itself to the stable API.

To some extent, that exists at the moment - many of the real abuses of the
CPython internals can be controlled by setting C defines. For the
particular feature that caused this discussion the majority of the uses can
be turned off by defining CYTHON_USE_EXC_INFO_STACK=0 and
CYTHON_FAST_THREAD_STATE=0. (There's still a few uses relating to
coroutines, but those too flags are sufficient to get Cython to build
itself and Numpy on Python 3.11a4).

Obviously it could still be better. But the desire to support PyPy (and
the beginnings of the limited API) mean that Cython does actually have
alternate "clean" code-paths for a lot of cases.

Hm... So maybe the issue is either with Cython's default settings (perhaps
traditionally it defaults to "as fast as possible but relies on internal
APIs a lot"?) or with the Cython settings selected by default by projects
*using* Cython?

I wonder if a solution during CPython's rocky alpha release cycle could be
to default (either in Cython or in projects using it) to the "not quite as
fast but not relying on a lot of internal APIs" mode, and to switch to
Cython's faster mode only once (a) beta is entered and (b) Cython has been
fixed to work with that beta?

This seems tempting – with the drawback that it would make Cython modules
less comparable between final and alpha/beta CPython releases. So users
would start reporting ghost performance regressions because it
(understandably) feels important to them that the slow-down they witness
needs to be resolved before the final release, and they just won't know
that this will happen automatically triggered by the version switch. :)

Feels a bit like car manufacturers who switch their exhaust cleaners on and
off based on the test mode detection.

More importantly, though, we'd get less bug reports during the alpha/beta
cycle ourselves, because things may look like they work but can still stop
working when we switch back to fast mode.

I'd rather make it more obvious to users what their intentions are. And
there is already a way to do that – the Limited API. (and similarly, HPy)

For Cython, support for the Limited API is still work in progress, although
many things are in place already. Getting it to work completely would give
users a simple way to decide whether they want to opt in for a) speed, lots
of wheels and adaptations for each CPython version, or b) less performance,
less hassle.

As it looks now, that switch can be done after the code generation, by
defining a simple C define in their build script. That also makes both
modes easily comparable. I think that is as good as it can get.

Stefan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/FXSNX7UCQWNXXC7OWG4LBLILAYXQEOUB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

2022-02-01 Thread Inada Naoki

On Wed, Feb 2, 2022 at 8:53 AM Thomas Caswell  wrote:
>
> I disagree with point (3).
>
> I think it would be better to discourage projects from including the output 
> of cython in their sdists.  They should either have cython as a build-time 
> requirement or provide built wheels (which are specific a platform and 
> CPython version).  The middle ground of not expecting the user to have cython 
> while expecting them to have a working c-complier is a very narrow case and I 
> think asking those users to install cython is worth the forward compatibility 
> for Python versions you get by requiring people installing from source to 
> re-cythonize.
>

I don't think having C compiler but not cython is the "narrow" case.
Requiring the latest Cython is installed is heavy additional dependency.
Note that we don't require the latest C compiler. Users may install C
compiler via tools like `dnf` or `apt`. But they can not install the
latest Cython via `dnf` or `apt`.

So this should be considered case-by-case basis.

* If the project can provide wheels for very wide platforms, stop
bundling C source is not a big problem.
* If the project can not provide wheels for some reason (*), they want
to bundle C source.
  * They can release source package more than a year.
  * Or they want to use "only public API" mode for better compatibility.

(*) For example, my "mysqlclient" library provide wheel only on
Windows because user may want to use own libmysqlclient, and I don't
want to maintain binary linking OpenSSL.

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3A4ZTNOSMJIDAMCVTIGRMIN6LXLTWOGV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python


On 2/02/22 12:32 pm, Christopher Barker wrote:


I can make a "thick" high performance wrapper, e.g. if I want to call an 
expensive C function on each item in a sequence, I can do that in Cython, removing a lot 
of the overhead of Python.


"Not as fast as possible" doesn't necessarily mean *slow*. Even using
the stable ABI, the code you get will still be a lot more efficient than
a Python wrapper.

Sure, but this thread is not just about users like me, that can choose 
the more stable way or the faster way, but specifically about numpy, 
which is going to use the fast way -- and we don't want to break that 
any more than absolutely necessary.


I'm skeptical about how much difference it would actually make. Numpy
gets its speed from tight loops in C and calling out to C and Fortran
libraries. None of that is affected by which CPython API is being
used.

In any case, if numpy explicitly chooses speed over compatibility, 
that's an issue between CPython and numpy, not CPython and Cython.


--
Greg


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UJOVLMM4OJVSGZAEVVTTNQJCNB4GWTYS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

On Tue, Feb 1, 2022 at 4:14 PM Stefan Behnel  wrote:

> Guido van Rossum schrieb am 02.02.22 um 00:21:
> > I wonder if a solution during CPython's rocky alpha release cycle could
> be
> > to default (either in Cython or in projects using it) to the "not quite
> as
> > fast but not relying on a lot of internal APIs" mode, and to switch to
> > Cython's faster mode only once (a) beta is entered and (b) Cython has
> been
> > fixed to work with that beta?
>
> This seems tempting – with the drawback that it would make Cython modules
> less comparable between final and alpha/beta CPython releases. So users
> would start reporting ghost performance regressions because it
> (understandably) feels important to them that the slow-down they witness
> needs to be resolved before the final release, and they just won't know
> that this will happen automatically triggered by the version switch. :)
>

It sounds like you are speaking from experience here, so I won't argue.

> Feels a bit like car manufacturers who switch their exhaust cleaners on
> and
> off based on the test mode detection.
>

That would be more like detecting benchmarks and doing something different.
In terms of the car manufacturing process, we're talking about testing next
year's model before production has started up yet. If the new model uses
more gas than last year's, that would be a problem that needs to be solved
before production starts, but what we seem to have with Cython is more like
the new model's doors don't open. :-)

It may be hard to imagine if you're working on Cython, which only exists
because of performance needs, but there are other things that people want
to test with the upcoming CPython release in addition to performance (are
the seats comfortable? do the controls for the moonroof work?), and given
the long dependency chains in modern apps and packages, people want to get
started on those things early. Until numpy builds with Cython for CPython
3.11, nobody can start testing scikit-learn with CPython 3.11, and that's
frustrating for the scikit-learn maintainers.

> More importantly, though, we'd get less bug reports during the alpha/beta
> cycle ourselves, because things may look like they work but can still stop
> working when we switch back to fast mode.
>

True, true. Nobody's perfect.

> I'd rather make it more obvious to users what their intentions are. And
> there is already a way to do that – the Limited API. (and similarly, HPy)
>

Your grammar confuses me. Do you want users to be clearer in expressing
their intentions?

> For Cython, support for the Limited API is still work in progress,
> although
> many things are in place already. Getting it to work completely would give
> users a simple way to decide whether they want to opt in for a) speed,
> lots
> of wheels and adaptations for each CPython version, or b) less
> performance,
> less hassle.
>

But until that work is complete, we're stuck with the unlimited API, right?
And by its own statements in a recent post here, HPy is still not ready for
all use cases, so it's also still a pipe dream.

> As it looks now, that switch can be done after the code generation, by
> defining a simple C define in their build script. That also makes both
> modes easily comparable. I think that is as good as it can get.
>

Do you have specific instructions for package developers here? I could
imagine that the scikit-learn maintainer (sorry to pick on you guys :-)
might not know where to start with this if until now they've always been
able to rely on either numpy wheels or building everything from source with
default settings.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DDM5KQL4OF3LFKT2Z3SDG2A2UNOFJYBQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python


Thomas Caswell schrieb am 01.02.22 um 23:15:

I think it would be better to discourage projects from including the output
of cython in their sdists.  They should either have cython as a build-time
requirement or provide built wheels (which are specific a platform and
CPython version).  The middle ground of not expecting the user to have
cython while expecting them to have a working c-complier is a very narrow
case and I think asking those users to install cython is worth the forward
compatibility for Python versions you get by requiring people installing
from source to re-cythonize.


I agree. Shipping the generated C sources was a very good choice as long as 
CPython's C-API was very stable and getting a build time dependency safely 
installed on user side was very difficult.


These days, it's the opposite way.

Stefan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KTWDJGHPQW7AIKDQQYV4IFHAKQZVXACL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python

Guido van Rossum schrieb am 02.02.22 um 01:43:

It may be hard to imagine if you're working on Cython, which only exists
because of performance needs, but there are other things that people want
to test with the upcoming CPython release in addition to performance

I know. Cython (and originally Pyrex) has come a long way from a tool to
get stuff done to a dependency that a large number of packages depend on.
Maintainer decisions these days are quite different from those 10 years
ago. Let alone 20.

Let's just try to keep things working in general, and fix stuff that needs
to be broken.

On Tue, Feb 1, 2022 at 4:14 PM Stefan Behnel wrote:

I'd rather make it more obvious to users what their intentions are. And
there is already a way to do that – the Limited API. (and similarly, HPy)

Your grammar confuses me. Do you want users to be clearer in expressing
their intentions?

Erm, sort of. They should be able to choose and express what they prefer,
in a simple way.

For Cython, support for the Limited API is still work in progress, although
many things are in place already. Getting it to work completely would give
users a simple way to decide whether they want to opt in for a) speed,
lots of wheels and adaptations for each CPython version, or b) less
performance, less hassle.

But until that work is complete, we're stuck with the unlimited API, right?
And by its own statements in a recent post here, HPy is still not ready for
all use cases, so it's also still a pipe dream.

Yes. HPy is certainly far from ready for anything real, but even for the
Limited API, it's still unclear whether it's actually complete enough to
cover Cython's needs. Basically, the API that Cython uses must really to be
able to implement CPython on top of itself. And at the same time interact
not with the reimplementation but with the underlying original, at the C
level. The C-API, and especially the Limited API, were never really meant
for that.

Do you have specific instructions for package developers here? I could
imagine that the scikit-learn maintainer (sorry to pick on you guys :-)
might not know where to start with this if until now they've always been
able to rely on either numpy wheels or building everything from source with
default settings.

It's not well documented yet, since the implementation isn't complete, and
so, a bunch of things simply won't work. I don't remember if the buffer
protocol is part of the Limited API by now, but last I checked it was still
missing, so the scikit-learn (or NumPy) people would be fairly unhappy with
the current state of affairs.

But it's mostly just passing "-DCYTHON_LIMITED_API" to your C compiler.
That's the part that will still work but won't do (yet) what you think.
Because then, you currently also have to define "-DPy_LIMITED_API=..." and
that's when your C compiler will get angry with you.

Stefan

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/2UFG7IPKR77HQG36BZAUEUDJJKIGBSLE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Please update Cython before introcuding C API incompatible changes in Python