andygrove commented on issue #6253:
URL: 
https://github.com/apache/datafusion-comet/issues/6253#issuecomment-7017135720

   I'm closing #6279 without merging. This records what it did, what it 
measured, and what is left, so the work can be picked up again. The branch is 
kept at `andygrove/datafusion-comet:fix-6253-window-memory` (head `b87b0dc964`, 
mergeable against main when it was closed).
   
   ## The gap
   
   DataFusion's window operators register no `MemoryConsumer`. `WindowAggExec`, 
which Comet uses for `PERCENT_RANK`, `CUME_DIST`, `NTILE` and any aggregate 
whose frame ends at `UNBOUNDED FOLLOWING` (including Spark's default frame for 
`agg(x) OVER (PARTITION BY k)`), buffers the task's whole input and 
concatenates it without telling the pool. DataFusion 56 (`branch-56` at 
`964f925`) is unchanged here: nothing under 
`datafusion/physical-plan/src/windows/` reserves memory or spills. Upstream 
spilling is apache/datafusion#22946, still open.
   
   ## What #6279 did
   
   - A new `CometWindowAggExec` 
(`native/core/src/execution/operators/window_agg.rs`, wired in at 
`planner/window.rs`) that evaluates the same `WindowExpr`s one window partition 
at a time. It emits each window partition as soon as the sorted input moves 
past it and keeps only the one that may continue. Without `PARTITION BY` it 
still buffers everything.
   - It reserves what it buffers with `RecordBatchMemoryCounter` (shared 
buffers counted once) and reserves the concatenated copy by counting only the 
buffers the copy does not share with its inputs. Per-batch 
`get_slice_memory_size` estimates over-charged batches sliced from one list 
array about tenfold.
   - It registers with the pool when it buffers its first batch, not in 
`execute()`. Registering earlier shrinks the sort's share under `fair_unified` 
while the sort is still reading.
   - A refused reservation says the operator cannot spill, that a window 
partition needs up to twice its size within the operator's share, and which 
settings to change (`spark.memory.offHeap.size`, then `greedy_unified`, then 
`spark.comet.exec.window.enabled=false`).
   
   Reservation alone was not enough. A build that only reserved the buffered 
batches failed `SUM(v) OVER (PARTITION BY k)` over 200,000 rows in 5-row window 
partitions under a 4 MB pool, because `WindowAggExec` holds the whole task 
input, not one window partition. The streaming emission is what makes the 
reservation usable.
   
   ## Measurements (standalone bench, release, 2M rows in 8192-row batches)
   
   - Within 5% of `WindowAggExec` for 10-row and 100,000-row window partitions 
and with no `PARTITION BY`. 6-15% slower with 1,000-row window partitions 
(about 0.5 ms per 2M rows), because it concatenates per batch.
   - 20M rows (480 MB) in 1,000-row window partitions: `WindowAggExec` peaked 
at 1,254 MB RSS, reserved 0 and emitted one 20M-row batch. `CometWindowAggExec` 
peaked at 480 MB (the input itself), reserved 609 KB at peak and emitted 
9,000-row batches.
   - Without `PARTITION BY` both peak at 1,090 MB, and `CometWindowAggExec` 
reserves 960 MB of it.
   
   ## Tests worth reusing
   
   - A Rust differential test against `WindowAggExec` over 20 seeds (random 
window partition sizes, random batch boundaries including empty batches, null 
keys, two key columns, with and without `PARTITION BY`), plus mutation-checked 
memory tests: no batch reservation, no copy reservation, no release after 
emitting, early registration, never continuing a window partition across 
batches, not merging a continued window partition.
   - `windows/window_whole_partition_batches.sql` (whole-partition functions 
over 7-row batches).
   - Two `CometWindowExecSuite` tests: many small window partitions under a 4 
MB pool pass and match Spark, and a window partition that does not fit fails 
the task with the `CometWindowAggExec cannot spill` hint.
   - Spark 4.1.3 `DataFrameWindowFunctionsSuite` and 
`DataFrameWindowFramesSuite` gave the same result on main and on the branch (74 
pass, the same 4 fail, all marked `IgnoreComet`).
   
   ## What was not done
   
   - The `fair_unified` TPC-DS window query with a pool small enough to make 
the sort spill, compared against main. It was asked for in review. The 
standalone bench does not cover whether the share dilution or the per-batch 
pool calls cost anything there.
   - `BoundedWindowAggExec` reserves nothing and is not covered. A `RANGE ... 
CURRENT ROW` frame keeps a whole group of peer rows, so a skewed `ORDER BY` 
value can still grow untracked. That is #7092.
   - Spilling. A window partition that exceeds the operator's share fails the 
task where Spark would spill. In on-heap mode the pool is unbounded, so nothing 
is refused there.
   
   ## Open question for whoever resumes this
   
   Comet carries its own copy of `WindowAggStream` and about ten delegated 
methods through every DataFusion upgrade. The alternatives are to offer the 
streaming-emission stream upstream as the no-spill first step of 
apache/datafusion#22946, or to wait for that issue to get a design. Two earlier 
upstream attempts (apache/datafusion#22947 and apache/datafusion#23207) were 
closed unmerged, and apache/datafusion#23207 only reserved while still 
buffering the whole input. As of 2026-11-17 I had not found an upstream design 
that emits completed window partitions early. Review feedback on #6279 
suggested merging the DataFusion 56 upgrade first and then reanalyzing what is 
still open here.
   
   Related DataFusion window work filed by comphead: apache/datafusion#25637 
(spilling coverage epic), and apache/datafusion#26537, apache/datafusion#26539 
and apache/datafusion#26541 (`BoundedWindowAggExec` buffering and per-partition 
cost).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to