andygrove commented on issue #6224: URL: https://github.com/apache/datafusion-comet/issues/6224#issuecomment-5850467670
This is reachable on main without #5613, but only through some paths. While `ExecutionMemoryPool.acquireMemory` waits for memory, it waits on the memory manager's lock but keeps holding the task memory manager's monitor. So `TaskMemoryManager.freePage`, which takes that monitor, blocks behind a parked acquire instead of racing it. `releaseExecutionMemory` doesn't take the monitor, and that is what `MemoryConsumer.freeMemory` and `CometTaskMemoryManager.releaseMemory` use. Under `greedy_unified`, which has no pool lock, one native thread releasing the task's last bytes fails another native thread's parked acquire. I reproduced this at the component level on Spark 4.1.3. Two threads shared one `CometTaskMemoryManager` over an off-heap `UnifiedMemoryManager`. When a release took the task's balance to zero, the parked `acquireMemory` threw `NoSuchElementException: key not found: 1`. When only part was released, it stayed parked, and it completed once the other task freed memory. Under `fair_unified` on main, the pool's mutex serializes native acquires and releases, and the shuffle allocator frees pages with `freePage`. That leaves only a JVM off-heap consumer that frees with `freeMemory`. Either way, the parked `try_grow` fails with an error that isn't `ResourcesExhausted`, so the operator can't spill and the task fails. I'm labelling this as a bug. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
