whr778 commented on issue #48137:
URL: https://github.com/apache/arrow/issues/48137#issuecomment-5911455971
We hit this on pyarrow 25.0.1 (macOS arm64, Python 3.12) through Hugging
Face `datasets` streaming parquet: ~176 configs read concurrently from 8 Python
threads. Every file is written and `main()` returns, but the process never
exits.
A native sample of the stuck process shows:
```
exit (libsystem_c)
__cxa_finalize_ranges
std::shared_ptr<arrow::internal::ThreadPool>::~shared_ptr()
arrow::internal::ThreadPool::~ThreadPool()
arrow::internal::ThreadPool::Shutdown(bool)
std::condition_variable::wait
```
with 8 pool workers (`ThreadPool::LaunchWorkersUnlocked` threads) idle in
`condition_variable::wait`, at 0% CPU, indefinitely. It reproduced 3/3 on
25.0.1 and does not reproduce on 26.0.0.dev323, which includes GH-48137's fix.
We could not bisect, since every available nightly postdates the fix, so this
is consistent with rather than proven to be GH-48137.
Would a backport to a 25.0.x patch release be possible? Many `datasets`
users on 25.x would hit this at the end of long streaming jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
