ranflarion opened a new pull request, #27576: URL: https://github.com/apache/datafusion/pull/27576
## Which issue does this PR close? - Closes #27575. ## Rationale for this change A partial aggregate that skips aggregation calls `convert_to_state` on every batch, and for `sum` `PrimitiveGroupsAccumulator` allocates a zeroed array per batch and reads it before writing it. That is an extra pass with any allocator and 34 page faults per call with jemalloc. Details and numbers per allocator are in the issue. ## What changes are included in this PR? `PrimitiveGroupsAccumulator::convert_to_state` computes `prim_fn(starting_value, value)` per row with `compute::unary` instead of `PrimitiveArray::from_value` followed by `compute::binary_mut`. The filter handling is unchanged, and the error for a shared buffer goes away with the scratch array it guarded. `cargo bench -p datafusion-functions-aggregate --bench sum`, the five `sum i64 convert state` cases, per 8,192-row batch: | | main | this PR | change | |---|---|---|---| | Apple M4 Pro, macOS system allocator | 2.68 to 2.85 µs | 0.80 to 0.98 µs | -66% to -71% | | AMD EPYC, Linux, glibc | 2.33 to 2.62 µs | 1.04 to 1.25 µs | -52% to -56% | ## What is the testing strategy for this PR? Existing tests cover it. `aggregate_skip_partial.slt` sets both probe thresholds to 0, so its `sum`, `min` and `max` queries, with nulls and `FILTER`, run through this function on every batch (dropping the null mask here fails six of them), and the aggregation fuzzer enables skip-partial at random and compares against the non-skipping result. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
