yo-han opened a new issue, #54536:
URL: https://github.com/apache/arrow/issues/54536
### Describe the bug, including details regarding any error messages,
version, and platform.
A short-lived process that reads a local parquet file through pandas and
then exits sometimes aborts with SIGABRT (exit 134), after all of its work and
output are done. stderr shows `terminate called without an active exception`
and `Fatal Python error: Aborted`, with no Python frame.
pandas opens a Python file handle for a local path and passes it to the
dataset reader, which wraps it in a `PyReadableFile`. A reader task can drop
the last reference on a ThreadPool worker thread while the interpreter is
finalizing. `OwnedRefNoGIL::~OwnedRefNoGIL` then calls `PyGILState_Ensure`,
CPython ends the thread with `pthread_exit`, the forced unwind passes a
noexcept destructor, and `std::terminate` is called.
**Reproduction**
```python
# write.py: run once
import pandas as pd
pd.DataFrame({"a": [1, 2], "b": [1.0, 2.0]}).to_parquet("p.parquet",
index=False)
```
```python
# read.py: run many times, a few in parallel
import pandas as pd, sys
pd.read_parquet(sys.argv[1], columns=["a", "b"])
print("ok")
```
```
python write.py
seq 3000 | xargs -P 4 -I{} sh -c 'python read.py p.parquet >/dev/null 2>&1
|| echo $?' | sort | uniq -c
```
Expected: no output (every run exits 0). Observed: `38 134`, that is 38 of
3000 runs abort.
The same loop with `pyarrow.parquet.read_table(sys.argv[1], columns=["a",
"b"]).to_pandas()` in place of `pd.read_parquet` gives no output, so 0 of 3000
runs abort. The file is then opened natively, and no Python object is held by
the reader.
**Environment**
- pyarrow 26.0.0 (pyarrow 25.0.1 aborts at the same rate)
- pandas 3.0.6, numpy 2.5.3
- CPython 3.12.12
- Debian 12 (bookworm), aarch64, in Docker
**Stack of the aborting thread (gdb)**
```
#0 __pthread_kill_implementation
#3 abort
#4 __gnu_cxx::__verbose_terminate_handler()
#6 std::terminate()
#7 __gxx_personality_v0
#9 _Unwind_ForcedUnwind
#10 __pthread_unwind
#12 __pthread_exit
#13 PyThread_exit_thread
#14 take_gil
#15 PyEval_RestoreThread
#16 PyGILState_Ensure
#17 arrow::py::OwnedRefNoGIL::~OwnedRefNoGIL()
#18 arrow::py::PyReadableFile::~PyReadableFile()
#19 std::_Sp_counted_base<...>::_M_release_last_use_cold()
#20
std::_Function_handler<arrow::Result<std::shared_ptr<arrow::io::RandomAccessFile>>(),
arrow::dataset::FileSource::FileSource(...)>
#21 arrow::dataset::FileSource::~FileSource()
#22 arrow::dataset::ParquetFileFragment::~ParquetFileFragment()
... on a ThreadPool worker thread (ThreadPool::LaunchWorkersUnlocked)
```
**Possible fix.** Skip the decref in `OwnedRefNoGIL`'s destructor when the
interpreter is finalizing (`Py_IsFinalizing()`), as other C++ extensions do.
Alternatively, release the scan's fragments on the calling thread before
`read_table` returns.
**Related.** ARROW-16597 has the same mechanism in Flight:
`PyThread_exit_thread` → `pthread_exit` while C++ code is on the stack, at
interpreter exit.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
