sunchao commented on PR #25428: URL: https://github.com/apache/datafusion/pull/25428#issuecomment-5819977641
@comphead @viirya @jayzhan211 Updated in [c17e315d0f](https://github.com/apache/datafusion/commit/c17e315d0fe71cfc1db2c5a376dd1f5e70e14ed2). I narrowed this to intermediate input sizing. Widening, the batch-memory estimator and boundary flushing, the write retry, and their associated helpers/tests are removed. The ordinary admission, read-ahead, batching, and reservation ownership code matches the base. Only the newer `AggregateSpill` path opts in; legacy aggregation retains its selection. A guard requiring uniform recorded maximum batch memory and batch-size limits prevents extra rewrites in a heterogeneous IPC regression. Case G remains capped at fan-in two and does not exercise sizing. The [description](https://github.com/apache/datafusion/pull/25428) now contains only exact base `714956b3` versus this head: SF10 `external_aggr` Q2, four partitions, 32/48/64 MiB. It reports exact aggregate spill counters, separate repartition spills, trimmed passes, intermediate rows/bytes, batches per file, and query-wide peak tracked disk. The 48 MiB runs exercise sizing across at least two intermediate rewrite levels. All intermediate files in these integer-key runs have exactly `ceil(rows / 8192)` batches. There are no widened passes in this revision. The report retains the non-improving 32 MiB work counters and the 48 MiB peak-disk increase, and qualifies differences in initial spill layouts. Clean timings show a modest 2.60% mean improvement at 48 MiB. The first 64 MiB series was 1.66% slower; a separate fixed confirmation was 0.17% slower with mixed blocks, so both are reported without a latency claim for that setting. Validation passed: 12,069 Rust tests, all 522 SQL logic files, strict all-target/all-feature Clippy, and the full lint suite. Dependencies and the lockfile are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
