AndreaBozzo commented on issue #50482:
URL: https://github.com/apache/arrow/issues/50482#issuecomment-6448135134
An early stop makes this deterministic, and `close()` avoids it, but
releasing the C stream export does not, because `ExportedArrayStream::Release`
(`cpp/src/arrow/c/bridge.cc`) drops its `shared_ptr` without calling
`reader->Close()`.
```python
import tempfile, fsspec
import pyarrow as pa, pyarrow.dataset as ds, pyarrow.fs as pafs,
pyarrow.parquet as pq
root = tempfile.mkdtemp()
for i in range(8):
pq.write_table(pa.table({"x": range(1000)}), f"{root}/part{i}.parquet",
row_group_size=100)
fs = pafs.PyFileSystem(pafs.FSSpecHandler(fsspec.filesystem("file")))
reader = ds.dataset(root, filesystem=fs,
format="parquet").scanner().to_reader()
reader.read_next_batch()
# reader.close() # with this line: clean exit
```
Linux x86_64, PyArrow 26.0.0, CPython 3.12.3, 10 fresh processes each: no
close, 10/10 fail at exit (aborts with `PyGILState_Release ... finalizing`, or
hangs); `reader.close()`, 10/10 clean; reading through
`RecordBatchReader._import_from_c_capsule(reader.__arrow_c_stream__())` and
closing that, 10/10 fail; then also `reader.close()` on the original, 10/10
clean. Not tested against #54467.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
