comphead commented on PR #6279: URL: https://github.com/apache/datafusion-comet/pull/6279#issuecomment-6345846784
Here are the DataFusion items I filed for window execution: Spilling and memory - apache/datafusion#25637 is the spilling coverage epic. It tracks `WindowAggExec` and `BoundedWindowAggExec` as non-spilling next to Spark and DuckDB. The upstream spill work for `WindowAggExec` is apache/datafusion#22946. `BoundedWindowAggExec` and the value functions, which relate to #7092 - apache/datafusion#26537 proposes memoizing `LAST_VALUE` and `NTH_VALUE` with `IGNORE NULLS` so `BoundedWindowAggExec` can prune its buffer. - apache/datafusion#26539: `LAST_VALUE` and `NTH_VALUE` with `IGNORE NULLS` keep the whole partition buffered. - apache/datafusion#26541: `BoundedWindowAggExec` has a high fixed cost per partition when partitions are small. Performance - apache/datafusion#26538: `NTH_VALUE(expr, -n) IGNORE NULLS` is quadratic in the partition size. - apache/datafusion#26540: `FIRST_VALUE`, `LAST_VALUE` and `NTH_VALUE` build a `ScalarValue` for every row. - apache/datafusion#26532 and apache/datafusion#26533 are the issue and the fix that made `LAST_VALUE ... IGNORE NULLS` linear. #26533 merged on 2026-10-21. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
