raulcd opened a new issue, #50591: URL: https://github.com/apache/arrow/issues/50591
### Describe the bug, including details regarding any error messages, version, and platform. While reviewing: - https://github.com/apache/arrow/pull/50409 I started investigating and I found that we could leak a reference when converting an iterator that raises an Exception at `ConvertToSequenceAndInferSize`. I created a test that currently fails: ```python def test_failing_iterator_does_not_leak(): # ConvertToSequenceAndInferSize builds an intermediate list. If the # iterator raises, that list and everything already stored in it leaks. import gc # Create an arbitrary long int that is probably not cached by the interpreter value = 10**20 def raising_iter(): for _ in range(5): yield value raise ValueError("boom") gc.collect() original_refcount = sys.getrefcount(value) with pytest.raises(ValueError, match="boom"): pa.array(raising_iter(), size=100) gc.collect() assert sys.getrefcount(value) == original_refcount ``` The issue is in `RETURN_IF_PYERROR();` error case where lst is not `Py_DECREF`: https://github.com/apache/arrow/blob/154962c364bc210c10ef16680f54a115b624b400/python/pyarrow/src/arrow/python/python_to_arrow.cc#L1260-L1276 The failing iterator case was detected here: - https://github.com/apache/arrow/pull/13764 ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
