koriyoshi2041 opened a new pull request, #26788: URL: https://github.com/apache/datafusion/pull/26788
Closes #26772. ## Problem `WindowAggExec` and grouped TopK materialize their final result and currently emit it as one `RecordBatch`. Large results therefore bypass `datafusion.execution.batch_size` and make one stream poll hand an oversized batch to downstream operators. ## Fix Carry the configured batch size into both streams, retain the materialized result, and emit zero-copy `RecordBatch::slice` chunks across successive polls. Output metrics are recorded per emitted chunk, and `WindowAggExec` releases its buffered input batches after computing the result. ## Tests - `cargo test -p datafusion-physical-plan grouped_topk_stream::tests --lib` - `cargo test -p datafusion-physical-plan window_agg_exec::tests --lib` - `cargo clippy -p datafusion-physical-plan --lib --tests -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` The regressions assert `4/4/2` and `2/2/1` output shapes, unchanged values, and per-chunk row/batch metrics. ## Risk Low. Result computation, ordering, and values are unchanged. Slicing is zero-copy; the behavioral change is limited to output batch boundaries and the matching output-batch metric count. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
