andygrove commented on PR #5998: URL: https://github.com/apache/datafusion-comet/pull/5998#issuecomment-5719753823
Triage note: #5027 overlaps with this on the UDF output path. It cuts a per-task Arrow child allocator backed by a non-spillable `MemoryConsumer` for `CometScalaUDFCodegen`'s output vectors, which accounts that one path more precisely than a root-level listener does, and it changes the `CometUDF` interface to pass the allocator through. Both PRs touch `CometUdfBridge.java`, and if both land unchanged those output buffers are charged twice. @peterxcli, worth deciding which layer owns those buffers. My instinct is that the root listener here covers everything by default and the per-task consumer in #5027 should replace it on the UDF path rather than stack on top, but I have not tried writing that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
