jackboyla commented on PR #54467: URL: https://github.com/apache/arrow/pull/54467#issuecomment-6321774715
I found a pending-I/O case while investigating huggingface/datasets#8793: after a Parquet prebuffer read error, the CPU pool can be idle while other reads still hold Python callbacks. Both threaded and nonthreaded decoding reproduce the fatal shutdown error on this PR's current head. I opened an extension against your branch: https://github.com/mikamikasuki/arrow/pull/1. It waits for the default CPU and I/O pools to be idle together, since tasks can submit work between pools. The new subprocess regressions fail on the CPU-only implementation and pass with the extension in full source builds. A reconstructed two-source Hub workload also reproduced the fatal error after 1.509 GB on the CPU-only build with unmodified Datasets. With both the Datasets iterator-ownership fix and this extension, two matching full-size runs exited cleanly; all output hashes matched. Would you be open to incorporating the extension into this PR? The test and implementation changes are one commit on top of your branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
