adriangb opened a new issue, #25604: URL: https://github.com/apache/datafusion/issues/25604
### Is your feature request related to a problem or challenge? `GroupsAccumulatorAdapter` runs every aggregate that has no `GroupsAccumulator` of its own, for example `covar_samp`, `approx_percentile_cont` and user-defined aggregates that implement only `Accumulator`. For each input batch, `invoke_per_accumulator` allocates new buffers: - `groups_with_rows`: the groups that have rows in the batch. - `offsets`: where the rows of each of those groups start. - `batch_indices`: the row indexes in group order, for the `take` kernel. The adapter drops these buffers at the end of the batch and allocates them again for the next batch. @alamb suggested this in https://github.com/apache/datafusion/pull/25123#discussion_r4066870146. ### Describe the solution you'd like Keep `groups_with_rows` and `offsets` on the adapter and reuse them for each batch. The adapter must charge the capacity that it keeps to its memory accounting (`GroupsAccumulator::size`), in the same way as it charges the per-group `indices` since https://github.com/apache/datafusion/pull/24858. It must release that memory when it emits all groups. `batch_indices` is different. `batch_indices.into()` moves the `Vec` into the `UInt32Array` that `take_arrays` and `get_filter_at_indices` read. To reuse it, the adapter must copy it into a new array for each batch, which costs about the same as the allocation that it saves. ### Describe alternatives you've considered Keep the current behavior. It is one allocation of each buffer for each batch, while the same batch calls `Accumulator::update_batch` once for each group that it touches. Thus the gain is probably small. ### Additional context Follow-up to https://github.com/apache/datafusion/pull/25123. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
