dashdartgit commented on issue #48137:
URL: https://github.com/apache/arrow/issues/48137#issuecomment-6656072738
Following up on the 2026-09-30 report above: we still see this stack on
**pyarrow 26.0.1** (released 2026-11-03), macOS 27 arm64, CPython 3.12, pandas
3.0.3.
A script that reads three small local parquet files with
`pd.read_parquet()`, prints one line and returns from `main()` hung at exit in
**11 of 1,000 runs on 26.0.1**. Every hang, sampled with `sample`, shows the
stack above (`exit` → `__cxa_finalize_ranges` → `~shared_ptr<ThreadPool>` →
`ThreadPool::Shutdown` → `condition_variable::wait`), with an idle pool worker
in `condition_variable::wait` and no `PyGILState_Ensure` frame, so as far as we
can tell it is not #50482's mechanism.
It needed all three of:
1. the Arrow CPU pool used (here, `pd.read_parquet` on local files);
2. stdout connected to a **pipe** (`subprocess.run(...,
capture_output=True)`): 14 hangs in 1,500 runs on a pipe, against 0 in 1,500
with stdout redirected to a file and 0 in 600 on a tty;
3. the process exiting within tens of milliseconds of its last read. Each of
these gave 0 in 600, beside about 10 in 600 for the unmodified script: a 50 ms
sleep before exit, `pa.set_cpu_count(1)`, `OMP_NUM_THREADS=1`,
`pq.read_table(path).to_pandas()`, or `pd.read_parquet(path,
filesystem=pyarrow.fs.LocalFileSystem())`.
Other versions, same harness (6 runs in parallel; a run counts as hung when
it is still alive 25 s after printing its last line): 21.0.0 15/800, 23.0.1
8/800, 24.0.0 about 1.7–2.5%, 26.0.1 11/1,000.
Minimal reproducer (16 hangs in 600 runs on 24.0.0, same harness):
```python
# setup.py — run once
import os
import pandas as pd
os.makedirs("data", exist_ok=True)
for k in range(3):
pd.DataFrame({"ticker": ["AAA"] * 2, "date": ["2023-01-03",
"2023-01-04"]}).to_parquet(f"data/f{k}.parquet")
```
```python
# repro.py — run many times, e.g.
# subprocess.run([sys.executable, "repro.py", "data"],
capture_output=True, timeout=25)
import sys, glob
import pandas as pd
for f in sorted(glob.glob(sys.argv[1] + "/*.parquet")):
pd.read_parquet(f)
print("DONE")
```
Happy to open a separate issue if that is more useful than a comment on a
closed one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
