thisisnic commented on issue #37798: URL: https://github.com/apache/arrow/issues/37798#issuecomment-5822138460
Closing as fixed in 14.0.0 by #37576. In 13.0.0 the R package's memory pool responded to a failed allocation by calling `gc()` directly on whatever thread the allocation happened on. During `collect()` on a dataset the allocations happen on Arrow worker threads, so under memory pressure R was being called from a non-R thread. That is undefined and shows up as a hang or crash, intermittently and more often with bigger data, which matches this report. #37576 routed that gc call through `SafeCallIntoR()`. I reproduced the mechanism on Linux by running this issue's reprex under an address-space limit so scan allocations fail, using the same R 4.3.1 and dplyr for both versions: - arrow 13.0.0: aborts on the first iteration every time. gdb shows an I/O worker thread inside `GcMemoryPool::GcAndTryAgain` calling into R via cpp11. - arrow 25.0.1: returns a clean `Out of memory: malloc of size N failed` error each time, never hangs. @NedJWestern if you still see this on a current release, please ping me here and I'll reopen. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
