andygrove commented on PR #5027: URL: https://github.com/apache/datafusion-comet/pull/5027#issuecomment-5719753152
Triage note: #5998 overlaps with this. It attaches an `AllocationListener` to the process-wide `CometArrowAllocator` root so that every JVM Arrow allocation — FFI export, broadcast coalescing, `CometSparkToColumnarExec`, the cached batch serializer, codegen output and the Python runner — is reserved from Spark's `TaskMemoryManager` in fixed blocks. That one is mine, and it is the first item of #5997. They are not the same change. This PR cuts a per-task child allocator for the UDF output path specifically and charges it to a non-spillable `MemoryConsumer`, which is more precise on that path than a root-level listener, and the `CometUDF` interface change has no counterpart in #5998. But both charge the same output buffers and both touch `CometUdfBridge.java`, so landing them unchanged would double-count. Could we work out which layer owns the UDF output buffers before either goes further? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
