comphead opened a new pull request, #27462: URL: https://github.com/apache/datafusion/pull/27462
## Which issue does this PR close? - Part of #26541. ## Rationale for this change In `Sorted` mode, `BoundedWindowAggExec` builds a key, a buffered batch and a state per window expression for every partition, and prunes them again. When partitions hold one or a few rows, this bookkeeping dominates. The issue has a reproducer. ## What changes are included in this PR? When every window expression is an aggregate, partitions that start and end inside an input batch are evaluated with `WindowExpr::evaluate` on their slice of the batch. For an aggregate this is the calculation `evaluate_stateful` does for a whole partition, without the key, buffer or state. Only the first and last partition of a batch are buffered, and the results of the others are spliced in between them. A stream with a `WindowStateObserver` keeps the current path, since it needs the state of every partition. | 1M rows, `target_partitions = 1`, `datafusion-cli` (`release-nonlto`) | `main` | PR | |---|---:|---:| | `max(v) OVER (PARTITION BY k ORDER BY i)`, one row per partition | 0.61 s | 0.33 s | | the same with 10 carried string columns | 0.93 s | 0.56 s | | `sum(v)` over `ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING`, one row per partition | 0.56 s | 0.28 s | In the `window_query_sql` benchmark, `window partition and order by, u64_wide, aggregate functions` goes from 443 ms to 318 ms, and the other cases stay within noise. ## What is the testing strategy for this PR? - A `window.slt` case at `batch_size = 4`, where a batch starts inside a buffered partition, holds a complete one and ends inside another. - The observer exactly-once test gets a partition inside a batch, which must still be observed. - A UDWF test pins the evaluator calls for a partition inside a batch. ## Are there any user-facing changes? No. Known limitation: non-aggregate window functions keep the current path. Their `evaluate` calls other `PartitionEvaluator` methods than the stream does, and evaluating each partition with `evaluate_stateful` made `row_number` and `lag` slower. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
