jayzhan211 commented on PR #25428:
URL: https://github.com/apache/datafusion/pull/25428#issuecomment-5825561317
The uniformity guard only looks at the input runs, but the intermediate
output is rebatched up to `batch_size`. When the runs are shorter than
`batch_size`, the new run is wider than its inputs, the final pass no longer
fits in F slots, and sizing adds a pass instead of removing one. With 9 runs of
64 rows at `batch_size = 128` (same pool formula as
`intermediate_merge_sizing_preserves_final_output`), sizing off rewrites 448
rows (7 runs) and sizing on rewrites 576 rows (3 runs, then 6).
Suggest trimming only when every run holds full batches, so merged batches
can't grow. This needs `max_batch_rows` recorded on `SortedSpillFile` at spill
time:
```diff
.all(|(file, limit)| {
file.max_record_batch_memory
== first.max_record_batch_memory
&& limit == first_limit
+ && file.max_batch_rows == *limit
});
```
Please also add the short-run case as a test that asserts sizing never
rewrites more rows than the unsized path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
