sunchao commented on PR #5038: URL: https://github.com/apache/datafusion-comet/pull/5038#issuecomment-5387092772
Merged, thanks @peterxcli ! I think later on we also need to bump Arrow to 59.1+ in order to get the allocation savings. In Arrow 58.4.0, its `CompressionContext` contains only an optional zstd compressor, which Comet never initializes because Arrow IPC buffer compression is disabled. The `FlatBuffer` builder is allocated separately on every batch. Moving this unused context outside the loop therefore saves no allocations today. Comet’s outer compression encoder is still recreated per block. Starting with Arrow 59.1, the context owns the reusable `FlatBuffer` builder. At that point, this change genuinely reduces repeated metadata-allocation work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
