comphead opened a new issue, #26540: URL: https://github.com/apache/datafusion/issues/26540
### Is your feature request related to a problem or challenge? In `BoundedWindowAggExec`, `FIRST_VALUE`, `LAST_VALUE` and `NTH_VALUE` produce one `ScalarValue` per output row. `NthValueEvaluator::evaluate` turns the selected row into a scalar with `ScalarValue::try_from_array` ([nth_value.rs#L472](https://github.com/apache/datafusion/blob/3ee3e394c19c97ccc7cca7ed3becb75cf364746a/datafusion/functions-window/src/nth_value.rs#L472)), and `StandardWindowExpr::evaluate_stateful` collects the scalars ([standard.rs#L213-L244](https://github.com/apache/datafusion/blob/3ee3e394c19c97ccc7cca7ed3becb75cf364746a/datafusion/physical-expr/src/window/standard.rs#L213-L244)) and turns them back into an array with `ScalarValue::iter_to_array` ([#L252](https://github.com/apache/datafusion/blob/3ee3e394c19c97ccc7cca7ed3becb75cf364746a/datafusion/physical-expr/src/window/standard.rs#L252)). That is cheap for primitive values but not for nested ones. Each struct scalar is a one-row slice of every child array, and `iter_to_array` concatenates all of them again. In `datafusion-cli`, 1M rows in 64-row partitions, with a four-field struct: ```sql SET datafusion.execution.target_partitions = 1; CREATE TABLE s AS SELECT value / 64 AS k, value AS i, value AS v, named_struct('a', value, 'b', concat('algo-', value), 'c', concat('sub-', value), 'd', value % 7 = 0) AS st FROM generate_series(1, 1000000); SELECT count(x) FROM (SELECT last_value(st) OVER (PARTITION BY k ORDER BY i) AS x FROM s); ``` | Function | bigint `v` | struct `st` | |---|---:|---:| | `last_value` | 0.10 s | 0.26 s | | `nth_value(..., 2)` | 0.10 s | 0.28 s | ### Describe the solution you'd like These functions always return one of the frame's rows, or NULL. If the `PartitionEvaluator` could report the index of that row instead of a scalar, `StandardWindowExpr` could collect the indices and build each output column with one `take`. A prototype of this, a `PartitionEvaluator` method that returns the row index for `LAST_VALUE` and a gather in `StandardWindowExpr` when the evaluator supports it, made `last_value` over a five-field struct with 4 rows per partition 2.1x faster in Comet's window benchmark, with identical output. ### Describe alternatives you've considered A faster `iter_to_array` for struct scalars would remove the concatenation but keep a slice and an allocation per row. ### Additional context The timings are from `datafusion-cli` built from 3ee3e394c with the `release-nonlto` profile on an Apple M5 Max. Found through Apache DataFusion Comet, where Spark's `last(struct(...), true)` over sorted clickstream rows spends about half of its window time on these scalars. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
