MagicLex commented on issue #50482:
URL: https://github.com/apache/arrow/issues/50482#issuecomment-6342723756
The same exit hang also happens when the scan goes through a
`pyarrow.fs.PyFileSystem` instead of a Python file object (macOS arm64, CPython
3.12.12).
```python
# child.py DIR, where DIR is an absolute path holding 4 Parquet files of
1000 int64 rows
import sys, fsspec
import pyarrow.dataset as ds, pyarrow.fs as pafs
fs = pafs.PyFileSystem(pafs.FSSpecHandler(fsspec.filesystem("file")))
# control: fs = pafs.LocalFileSystem()
print(ds.dataset(sys.argv[1], filesystem=fs,
format="parquet").to_table().num_rows, flush=True)
```
Result on pyarrow 26.0.0, fresh processes, 8 in parallel, hung = printed the
count and still alive after 20 s: 15/200 hung with `PyFileSystem`, 0/200 with
`LocalFileSystem`.
Main thread of a hung process (`sample`):
```
exit
std::shared_ptr<arrow::internal::ThreadPool>::~shared_ptr
arrow::internal::ThreadPool::~ThreadPool()
arrow::internal::ThreadPool::Shutdown(bool)
```
PR #54467 likely covers this path; not tested.
Workaround: scan with a native filesystem (`LocalFileSystem`, or
`SubTreeFileSystem` over it) instead of a Python-implemented one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
