[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: New changeset 32f0c8271706550096c454eb512450b85fbfc320 by Victor Stinner in branch 'main': bpo-45459: Use type names in the internal C API (GH-31669) https://github.com/python/cpython/commit/32f0c8271706550096c454eb512450b85fbfc320 --

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: New changeset 0b63215bb152c06404cecbd5303b1a50969a9f9f by Victor Stinner in branch 'main': bpo-45459: Fix PyModuleDef_Slot type in the limited C API (GH-31668) https://github.com/python/cpython/commit/0b63215bb152c06404cecbd5303b1a50969a9f9f --

[issue45459] Limited API support for Py_buffer

Change by STINNER Victor : -- pull_requests: +29788 pull_request: https://github.com/python/cpython/pull/31669 ___ Python tracker ___

[issue45459] Limited API support for Py_buffer

Change by STINNER Victor : -- pull_requests: +29787 pull_request: https://github.com/python/cpython/pull/31668 ___ Python tracker ___

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: I close again the issue, the C API should now be fine :-) -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: New changeset 042f31da552c19054acd3ef7bb6cfd857bce172b by Victor Stinner in branch 'main': bpo-45459: C API uses type names rather than structure names (GH-31528) https://github.com/python/cpython/commit/042f31da552c19054acd3ef7bb6cfd857bce172b --

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: New changeset ec091bd47e2f968b0d1631b9a8104283a7beeb1b by Victor Stinner in branch 'main': bpo-45459: Add pytypedefs.h header file (GH-31527) https://github.com/python/cpython/commit/ec091bd47e2f968b0d1631b9a8104283a7beeb1b --

[issue45459] Limited API support for Py_buffer

Change by STINNER Victor : -- pull_requests: +29663 pull_request: https://github.com/python/cpython/pull/31539 ___ Python tracker ___

[issue45459] Limited API support for Py_buffer

Change by STINNER Victor : -- pull_requests: +29652 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/31528 ___ Python tracker ___

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: > Include/object.h:109:3: warning: redefinition of typedef 'PyObject' is a C11 > feature [-Wtypedef-redefinition] Oh. I already met this error :-( That's why I proposed in GH-31201 to move all forward declarations at the top of Python.h to solve such

[issue45459] Limited API support for Py_buffer

Change by STINNER Victor : -- pull_requests: +29651 pull_request: https://github.com/python/cpython/pull/31527 ___ Python tracker ___

[issue45459] Limited API support for Py_buffer

Benjamin Peterson added the comment: clang doesn't like the typedef forward-decl: In file included from ../cpython/Modules/_ctypes/_ctypes.c:108: In file included from ../cpython/Include/Python.h:43: ../cpython/Include/object.h:109:3: warning: redefinition of typedef 'PyObject' is a C11

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: pmp-p: > There's some side effects with "buffer.h" inclusion in Panda3D when building > againt 3.11a5, project manager concerns are here > https://github.com/python/cpython/pull/29991#issuecomment-1031731100 Thanks for the report. It has been fixed. I close

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: New changeset 66b3cd7063322a9f5c922a97bbd06fdb9830 by Victor Stinner in branch 'main': bpo-45459: Rename buffer.h to pybuffer.h (#31201) https://github.com/python/cpython/commit/66b3cd7063322a9f5c922a97bbd06fdb9830 --

[issue45459] Limited API support for Py_buffer

Change by STINNER Victor : -- pull_requests: +29372 stage: resolved -> patch review pull_request: https://github.com/python/cpython/pull/31201 ___ Python tracker ___

[issue45459] Limited API support for Py_buffer

Change by rdb : -- nosy: +rdb ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: > There's some side effects with "buffer.h" inclusion in Panda3D when building > againt 3.11a5, project manager concerns are here > https://github.com/python/cpython/pull/29991#issuecomment-1031731100 Copy of rdb's message: """ This change broke our project

[issue45459] Limited API support for Py_buffer

pmp-p added the comment: There's some side effects with "buffer.h" inclusion in Panda3D when building againt 3.11a5, project manager concerns are here https://github.com/python/cpython/pull/29991#issuecomment-1031731100 -- ___ Python tracker

[issue45459] Limited API support for Py_buffer

Change by pmp-p : -- nosy: +pmpp ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: Thanks for the review, Petr! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue45459] Limited API support for Py_buffer

miss-islington added the comment: New changeset f66c857572a308822c70fd25e0197b6e0dec6e34 by Christian Heimes in branch 'main': bpo-45459: Add Py_buffer to limited API (GH-29991) https://github.com/python/cpython/commit/f66c857572a308822c70fd25e0197b6e0dec6e34 -- nosy:

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: Thanks for the investigation. I didn't know about C99 Standard 6.7.8.21. That's a useful and sensible extension to the language. In my opinion it is neither useful to extend the Py_buffer struct with a version tag nor to force users to allocate the struct

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: Example: --- struct Point { int x; int y; int z; }; int main() { struct Point p = {1}; return p.y; } --- gcc -O0 produces this machine code which sets p.y to 0 and p.z to 0: --- Dump of assembler code for function main: 0x00401106 <+0>:

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: > The C language sets other members to 0/NULL with this syntax, no? No, they are not set to 0/NULL. https://en.wikipedia.org/wiki/Uninitialized_variable -- ___ Python tracker

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: > Py_buffer data = {NULL, NULL}; > The code initializes Py_buffer.buf and Py_buffer.obj as NULL. The remaining > fields are whatever random values happens to be on the C stack. The C language sets other members to 0/NULL with this syntax, no? --

[issue45459] Limited API support for Py_buffer

Petr Viktorin added the comment: The current struct is also likely to continue covering most future uses. If we decide to add PyBufferEx functions but continue providing the current ones (with the current struct), most users won't be affected. (But it'll be a bit more work for us than

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: The Py_buffer struct has stayed the same for over a decade and since Python 2.6.0 and 3.0.0. It is unlikely that it has to be changed in the near future. -- ___ Python tracker

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: In Python 3.5, I decided to rename the public "PyMemAllocator" structure to PyMemAllocatorEx when I added a new "calloc" member. C extensions using "PyMemAllocator" fail to build to force developers to set the calloc member. IMO it's unfortunate to have to

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: I thought of a version field, too. In the end it is going to cause more work and trouble than it would benefit us. Stack-allocated Py_buffer's are typically initialized with Py_buffer data = {NULL, NULL}; . The code initializes Py_buffer.buf and

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: Would it make sense to add a "version" member to the structure? It would allow to support an old stable structure for the stable ABI and a new structure with other changes. The problem is how to initalize the version member. On Windows, many structures have

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: After some consideration I also agree with Antoine. The Py_buffer API has been around for a long time without any changes to the Py_buffer struct. It is unlikely that the struct will ever change. I have created a new PR that exposes Py_buffer struct,

[issue45459] Limited API support for Py_buffer

Change by Christian Heimes : -- pull_requests: +28214 pull_request: https://github.com/python/cpython/pull/29991 ___ Python tracker ___

[issue45459] Limited API support for Py_buffer

Petr Viktorin added the comment: Antoine has a good point. We can freeze the Py_buffer struct. If it needs to be extended in the future, it'll need a new set of functions and names -- and perhaps a versioning scheme. We'll know more about the problem when/if it comes up. --

[issue45459] Limited API support for Py_buffer

Alex Gaynor added the comment: I am someone who is interested in having this, but FWIW my motivation is slightly more narrow, I only really need abi3-friendly buffer support with contiguous 1d buffers. Not sure if there'd be interest in doing a smaller version before figuring out the entire

[issue45459] Limited API support for Py_buffer

Antoine Pitrou added the comment: I would advocate: * expose the Py_buffer struct fully * expose the various PyBUF_* constants * expose at least PyObject_GetBuffer() and PyBuffer_Release() The rest is optional. -- ___ Python tracker

[issue45459] Limited API support for Py_buffer

Antoine Pitrou added the comment: Py_buffer *is* an ABI, and it hasn't changed from the start. Of course you can still try to collect feedback from third-party projects, but there is a very high probability that it won't need to change in the near future. --

[issue45459] Limited API support for Py_buffer

Change by Erlend E. Aasland : -- nosy: +erlendaasland ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue45459] Limited API support for Py_buffer

Petr Viktorin added the comment: No. Limited API is generally not as performant as the non-limited one. It is even documented this way: https://docs.python.org/3/c-api/stable.html#limited-api-scope-and-performance We should not make it *much* slower, but code that can take advantage of

[issue45459] Limited API support for Py_buffer

Serhiy Storchaka added the comment: That would be an unfair advantage. If we want people to use the limited API we should not make it much slower than the non-limited API. -- ___ Python tracker

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: CPython internals can still use allocation on the stack. Only stable ABI extensions have to use allocation on the heap. -- ___ Python tracker

[issue45459] Limited API support for Py_buffer

Serhiy Storchaka added the comment: I do not like requirement to allocate Py_buffer on the heap. It adds an overhead. Common case in CPython code is: Py_buffer view; void *buf; Py_ssize_t len; PyObject_GetBuffer(obj, , PyBUF_SIMPLE); buf = view.buf; len = view.len; // no other fields are

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: A consumer will use the APIs: --- Py_buffer *view; int ndim; const char *format; const Py_ssize_t *shape, *strides, *suboffsets; void *buf; view = PyBuffer_New(); PyObject_GetBuffer(obj, view, flags); ndim = PyBuffer_GetLayout(, , , ); buf =

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: All memory is owned by the exporter object. The exporter (aka producer) is the Python type that implements Py_bf_getbuffer and Py_bf_releasebuffer. In majority of cases the exporter doesn't have to set shape, strides, and suboffsets. They are used in

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: Maybe a PEP is needed to collect usages of the Py_buffer API and check if the ABI is future proof. A PEP may help to discuss with other projects which currently consume this API. I suggest to start with the smallest possible API and then slowly extend it.

[issue45459] Limited API support for Py_buffer

Petr Viktorin added the comment: > I recommend to get feedback from NumPy, Pillow, and Cython devs first. Could you split this into two PRs: one to add the new API, and another to add things to the limited set? There's no rush to add it to the limited API, esp. since it can't be tested

[issue45459] Limited API support for Py_buffer

Serhiy Storchaka added the comment: Py_buffer is often used for handling arguments if the function supports bytes, bytearray and other bytes-like objects. For example bytes.partition(). Any additional memory allocation would add significant overhead here. bytes.join() creates Py_buffer for

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: CC Antoine for his expertise of the buffer protocol Opaque Py_Buffer and PyObject structs will require a different approach and prevent some optimizations. The consumer will have to malloc() a Py_buffer struct on the heap. In non-trivial cases the

[issue45459] Limited API support for Py_buffer

Change by Christian Heimes : -- keywords: +patch pull_requests: +27306 stage: -> patch review pull_request: https://github.com/python/cpython/pull/29035 ___ Python tracker

[issue45459] Limited API support for Py_buffer

Serhiy Storchaka added the comment: ndim is not known before calling PyObject_GetBuffer(), so we will need a new API which combines PyObject_GetBuffer() and PyBuffer_New(). -- ___ Python tracker

[issue45459] Limited API support for Py_buffer

Christian Heimes added the comment: IIRC shape, strides, and suboffsets are all arrays of ndims length. We could optimize allocation if we would require users to specify the value of ndims and don't allow them to change the value afterwards. PyBuffer_New(int ndims) then would allocate view

[issue45459] Limited API support for Py_buffer

Serhiy Storchaka added the comment: shape is a pointer to array of Py_ssize_t of size ndim. array and memoryview do a trick to avoid memory allocation, but _testbuffer.ndarray allocates it dynamically in the heap. We can add a small static buffer in Py_buffer to avoid additional memory

[issue45459] Limited API support for Py_buffer

STINNER Victor added the comment: Py_buffer.shape requires a Py_ssize_t* pointer. It's not convenient. For example, the array module uses: static int array_buffer_getbuf(arrayobject *self, Py_buffer *view, int flags) { ... if ((flags & PyBUF_ND)==PyBUF_ND) { view->shape =

[issue45459] Limited API support for Py_buffer

New submission from Christian Heimes : Currently all APIs related to Py_buffer are excluded from the limited API. It's neither possible to use Py_buffer from a C extension with limited API nor is it possible to define heap types with buffer support using the stable ABI. The lack of Py_buffer