andygrove commented on PR #5027: URL: https://github.com/apache/datafusion-comet/pull/5027#issuecomment-5850486314
I closed #5998 after finding that charging JVM Arrow memory to Spark, and refusing on a short grant, turns native spills into task failures. This PR refuses the same way: `onPreAllocation` throws `OutOfMemoryException` when `acquireMemory` comes back short. `NativeMemoryConsumer.spill` returns 0, and native operators keep reserving until `try_grow` fails. So by the time a task is under pressure, native has already filled its share. The JVM allocation for the next batch asks just before native does, and that allocation is the one refused. In #5998's repro, at `local[4]` with 128m off-heap, a sort and a native shuffle over a Comet-cached table failed with `Unable to reserve ... got 0` in all four tasks. Both spilled normally with the refusal switched off. CI can't see this, because `CometTestBase` runs a 2 GiB pool. UDF output goes straight into native operators that could have spilled instead, so I'd expect the same failure here. Could the consumer record the allocation without refusing it? Or refuse only when nothing native in the task can spill? For the rest of the JVM Arrow memory, I ended up with #6250: it reports the figures in the memory usage log and doesn't gate on them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
