jbonofre commented on issue #246: URL: https://github.com/apache/arrow-java/issues/246#issuecomment-6744464655
The crash files that the RC workflow uploads since #1428 show the cause of the crashes in this test class. Details are in #1455. `NativeMemoryPool#close()` calls `releaseMemoryPool`, which immediately deletes the native `ReservationListenableMemoryPool` and the global reference to the Java listener. Scanners get the pool as a raw `arrow::MemoryPool*` through `ScannerBuilder::Pool`. After a scan fails, Arrow C++ can still run work for it on its thread pools: Parquet file opens, read-ahead, and destroying readers whose buffers came from the pool. That work then calls `Allocate` or `Free` on the deleted pool. Closing the scanner first doesn't help, because the close doesn't wait for that work. I only reproduced the failed-scan case, but a scanner closed before the end of the data may leave work running too. In `testErrorThrownFromReservationListener`, this crashed the test JVM in about 1 in 8 Binaries runs on `main`. #1457 stops that test from closing the pool, but the library problem remains. The original failure here (`expected:<8194> but was:<8193>` in `testDirectReservationListener`) looks like a reservation update arriving after the scan returned, which could have the same cause, but I haven't confirmed that. A fix in the library could be: - `releaseMemoryPool` marks the pool closed instead of deleting it. - After close, `Allocate` and `Reallocate` return an error status without calling the listener, and `Free` returns the memory to the parent pool without calling it. Today a failed listener call in `Impl::Free` hits `ARROW_LOG(FATAL)`. - The listener's global reference is deleted and the native object freed only once no call can still be in progress, for example with a count of in-flight calls. Waiting for `bytes_allocated()` to reach 0 isn't enough, since an allocation can start after the check. Leaking the small native object is a simpler option. Making a scanner's close wait for its outstanding work would also fix it, but I don't think the Arrow C++ scanner API offers a way to wait for that work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
