On 03. 06. 21 1:43, Victor Stinner wrote:
Hi,
What do you think of promoting the pythoncapi_compat project that I'm
introducing below in the "C API: Porting to Python 3.10" section of
What's New In Python 3.10?
Should this project be moved under the GitHub psf organization to have
a more "future proof" URL?
If you ask me, no.
I think we should aim to not break the C API as often. Rather than
promoting a tool to solve problems, I would much rather not create the
problems in the first place.
Sure, supporting HPy or per-interpreter GIL will be good when they're
ready to use. But at this point, they are *experiments*. I do not think
it's worth breaking the API for existing users, which get no benefit
from the changes, to run these experiments.
I would like to promote this project to prepare C extensions
maintainers for the following incompatible C API change (Py_TYPE) that
I would like to push into Python 3.11:
https://github.com/python/cpython/pull/26493
(Py_REFCNT was already converted to a static inline function in Python 3.10.)
I already made this Py_TYPE change in Python 3.10, but I had to revert
it since it broke too many projects. Since last year, I upgraded most
of these broken projects, I created the pythoncapi_compat project, and
I succeeded to use upgrade_pythoncapi.py script and copy the
pythoncapi_compat.h header file in multiple C extensions.
C extensions written with Cython are not affected. I already fixed
Cython last year to emit code compatible with my incoming incompatible
change. If it's not done yet, you only have to regenerate the C files
using a recent Cython version.
--
I wrote a new script which adds Python 3.10 support to your C
extensions without losing Python 2.7 support:
https://github.com/pythoncapi/pythoncapi_compat
To add Python 3.10 support to your C extension, go to its source
directory and run:
/path/to/upgrade_pythoncapi.py .
It upgrades all C files (.c) in the current directory and
subdirectories. For example, it replaces "op->ob_type" with
"Py_TYPE(op)". It creates an ".old" copy of patched files.
Use the -o option to select operations:
* -o Py_TYPE: only replace "obj->ob_type" with "Py_TYPE(obj)".
* -o all,-PyMem_MALLOC: run all operations, but don't replace
PyMem_MALLOC(...) with PyMem_Malloc(...).
--
The upgrade_pythoncapi.py script relies on the pythoncapi_compat.h
header file that I wrote to provide recent Python 3.9-3.11 C functions
on old Python versions. Examples: Py_NewRef() and
PyThreadState_GetFrame(). Functions are implemented as simple static
inline functions to avoid requiring to link your extension to a
dynamic library.
You can already use the new Py_NewRef() and Py_IsNone() Python 3.10
functions in your projects without losing support for Python 2.7-3.9!
--
The script also replaces "frame->f_back" with "_PyFrame_GetBackBorrow(frame)".
The _PyFrame_GetBackBorrow() function doesn't exist in the Python C
API, it's only provided by pythoncapi_compat.h to ease the migration
of C extensions. I advise you to replace _PyFrame_GetBackBorrow()
(borrowed reference) with PyFrame_GetBack() (strong reference).
--
This project is related to my PEP 620 "Hide implementation details
from the C API" which tries to make the C API more abstract to later
allow to implement new optimization in CPython and to make other
Python implementations like PyPy faster when running C extensions.
Article on the creation of the pythoncapi project:
https://vstinner.github.io/pythoncapi_compat.html
The main drawback of this project is that it uses regular expressions
to parse C code. Such "parser" can miss C code which has to be patched
manually. In my experience, additional manual changes are really rare
and take less than 1 minute on a very large C extension like numpy. >
--
This project only targets extension modules written in C by using
directly the "Python.h" API. I advise you to use Cython or HPy to no
longer be bothered with incompatible C API changes at every Python
release ;-)
* https://cython.org/
* https://hpy.readthedocs.io/
I hope that my script will facilitate migration of C extensions to HPy.
Victor
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/SY4X4MCIKIHDNBQPJ2JIBCP42D3LFPNP/
Code of Conduct: http://python.org/psf/codeofconduct/
