ranflarion opened a new issue, #27575: URL: https://github.com/apache/datafusion/issues/27575
### Is your feature request related to a problem or challenge? Once a partial aggregate skips aggregation, it calls `convert_to_state` on every input batch. `PrimitiveGroupsAccumulator` (`sum`, primitive `min` and `max`, the bit aggregates) implements it by allocating an array of `starting_value` and folding the input into it with `binary_mut`, which costs an allocation and a second pass per batch. For `sum` the starting value is 0, so that array is a zeroed allocation. jemalloc serves a zeroed 64 KiB allocation (8,192 `i64`s) with pages the kernel has not populated yet, and `binary_mut` reads each slot before writing it, so every call takes two page faults per 4 KiB page, 34 in all. `sum(i64)` `convert_to_state` on 8,192-row batches, ns per row per thread, on a 16 vCPU AMD EPYC running Linux: | threads | allocator | today | one pass | |---|---|---|---| | 1 | glibc | 0.29 | 0.12 | | 1 | mimalloc | 0.31 | 0.13 | | 1 | jemalloc | 7.26 | 0.14 | | 16 | glibc | 0.54 | 0.26 | | 16 | mimalloc | 0.56 | 0.28 | | 16 | jemalloc | 53.14 | 0.33 | ### Describe the solution you'd like Compute each row's state in one pass, `prim_fn(starting_value, value)` with `compute::unary`. It gives the same values and nulls without the scratch array. ### Describe alternatives you've considered Returning the input with the filter folded into its nulls, as `avg` does when the sum type equals the input type, would skip the pass entirely. That is only exact when `prim_fn(starting_value, v)` is `v`, which `PrimitiveGroupsAccumulator` cannot know (a float `sum` turns `-0.0` into `0.0`), so it would need a new constructor option. One pass removes the allocation without an API change. ### Additional context Found while profiling a Spark accelerator built on DataFusion that links jemalloc. With every grouping key distinct, over 91M rows on 16 threads, the skipping partial aggregate's `sum(a)` took 7.3 s of `elapsed_compute`, against 0.6 s for `count(*)` and 0.8 s for `min(a)`. ### Willingness to contribute I plan to contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
