fboudra opened a new pull request, #51375: URL: https://github.com/apache/arrow/pull/51375
### Rationale for this change https://github.com/apache/arrow/issues/50398 https://github.com/apache/arrow/issues/50829 ### What changes are included in this PR? Adds Py_LIMITED_API=0x030B0000 as a PUBLIC compile definition on arrow_python (inherited by parquet_encryption and flight) and passes --limited-api to Cython. From here on the tree only builds the limited API way, and the pyarrow Python layer requires it: under the limited API Cython emits heap types for cdef classes, and the ABC mixin registration pyarrow uses (for example KeyValueMetadata.get = Mapping.get followed by Mapping.register) needs heap types. A full API build of this tree compiles but fails at import. The PR carries the fixes this required: - Refcount semantics: PyWeakref_GetObject (the cp311 replacement for PyWeakref_GetRef) returns a borrowed reference, or Py_None when the referent is dead. PyExtensionType::GetInstance treated the result as a new reference, which underflowed the type instance refcount and caused the non-deterministic segfaults in test_pandas. It now INCREFs the alive result and treats Py_None as the dead case. - ABC bases removed from cdef classes (ABCMeta tp_new and PyType_Spec heap types on CPython >= 3.14); membership is done by mixin assignment and ABC.register in the Python layer. - PyObject_GetBuffer in common.h passes PyBUF_READ | PyBUF_STRIDES and checks the error. - Vendored pythoncapi_compat.h: skip the PyUnicodeWriter shims under the limited API (the real functions are stable since 3.3), and reimplement PyLong_GetSign without _PyLong_Sign. - Py_REFCNT is only exported as a stable symbol since 3.14, so libarrow_python and the Cython modules read the field directly (PyRawRefCnt in python_test.cc, a small limited_api_compat.h for the Cython modules). - DecimalFromString no longer uses the 's#' format specifier, which CPython 3.11 rejects with a SystemError; it builds the string with PyUnicode_FromStringAndSize and calls the constructor with PyObject_CallFunctionObjArgs. - Cython modules are named .abi3.so on POSIX (Windows keeps .pyd) so one cp311-abi3 wheel imports on every supported interpreter. - The PyBuffer invalid-input test uses a fresh empty list instead of Py_None, because CPython >= 3.14 immortal objects make the raw refcount assertion meaningless on Py_None. - One unrelated fix that was sitting in the series: the vendored tz database lookup in cpp/src/arrow/vendored/datetime/tz.cpp now honors the TZDB environment variable, matching the upstream date library. Verification: full pyarrow test suite on CPython 3.13 and 3.14, identical results on both (7522 passed, 889 skipped, 15 xfailed, 1 xpassed). The 12 failures and 4 errors are environmental and occur on unmodified upstream too (a pip tzdata wheel with non-standard TZif files, missing lz4 codec and test data in the local build). ### Are these changes tested? Yes ### Are there any user-facing changes? No ### Was AI used for this PR? **PR code and description written by:** - [x ] Human - [ ] AI **Reviewed before submission by:** - [ x] Human - [ ] AI - [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
