This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 826ac37440 GH-50591: [Python] Fix reference in
ConvertToSequenceAndInferSize when an iterator raises on array conversion
(#50594)
826ac37440 is described below
commit 826ac37440f0247769de1977e06956d49e710e05
Author: Raúl Cumplido <[email protected]>
AuthorDate: Wed Jul 22 13:36:59 2026 +0200
GH-50591: [Python] Fix reference in ConvertToSequenceAndInferSize when an
iterator raises on array conversion (#50594)
### Rationale for this change
We currently leak the list and items in the case of an iterator raising an
Exception and size is present.
### What changes are included in this PR?
Use `OwnedRef` instead of relying on manual `Py_DECREF` in order to avoid
exception scenario returning early in `RETURN_IF_PYERROR`.
### Are these changes tested?
Yes, I've created a new test that fails previously to the fix and passes
after the fix
### Are there any user-facing changes?
No
* GitHub Issue: #50591
Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
---
python/pyarrow/src/arrow/python/python_to_arrow.cc | 4 ++--
python/pyarrow/tests/test_convert_builtin.py | 23 ++++++++++++++++++++++
2 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/python/pyarrow/src/arrow/python/python_to_arrow.cc
b/python/pyarrow/src/arrow/python/python_to_arrow.cc
index 47290a7e7f..5b83875813 100644
--- a/python/pyarrow/src/arrow/python/python_to_arrow.cc
+++ b/python/pyarrow/src/arrow/python/python_to_arrow.cc
@@ -1259,6 +1259,7 @@ Status ConvertToSequenceAndInferSize(PyObject* obj,
PyObject** seq, int64_t* siz
OwnedRef iter_ref(iter);
PyObject* lst = PyList_New(n);
RETURN_IF_PYERROR();
+ OwnedRef lst_ref(lst);
for (i = 0; i < n; i++) {
PyObject* item = PyIter_Next(iter);
if (!item) {
@@ -1270,10 +1271,9 @@ Status ConvertToSequenceAndInferSize(PyObject* obj,
PyObject** seq, int64_t* siz
}
// Shrink list if len(iterator) < size
if (i < n && PyList_SetSlice(lst, i, n, NULL)) {
- Py_DECREF(lst);
RETURN_IF_PYERROR();
}
- *seq = lst;
+ *seq = lst_ref.detach();
*size = std::min<int64_t>(i, *size);
}
return Status::OK();
diff --git a/python/pyarrow/tests/test_convert_builtin.py
b/python/pyarrow/tests/test_convert_builtin.py
index c10ae0f62b..bb2813f3b5 100644
--- a/python/pyarrow/tests/test_convert_builtin.py
+++ b/python/pyarrow/tests/test_convert_builtin.py
@@ -128,6 +128,29 @@ def test_failing_iterator():
pa.array((1 // 0 for x in range(10)), size=10)
+def test_failing_iterator_does_not_leak():
+ # GH-50591
+ import gc
+ import sys
+
+ # Create an arbitrary long int that is hopefully not cached by the
interpreter
+ value = 10**20
+
+ def raising_iter():
+ for _ in range(5):
+ yield value
+ raise ValueError("boom")
+
+ gc.collect()
+ original_refcount = sys.getrefcount(value)
+
+ with pytest.raises(ValueError, match="boom"):
+ pa.array(raising_iter(), size=100)
+
+ gc.collect()
+ assert sys.getrefcount(value) == original_refcount
+
+
class ObjectWithOnlyGetitem:
def __getitem__(self, key):
return 3
