goutamadwant opened a new pull request, #24193: URL: https://github.com/apache/datafusion/pull/24193
## Which issue does this PR close? - Closes #24044. ## Rationale for this change `RepartitionExec` coalesces small batches on the producer side until the configured batch size is reached or all input senders finish. An unbounded input may never finish, so an available partial batch can be withheld indefinitely. This prevents an incremental query from emitting rows that are ready for downstream processing. ## What changes are included in this PR? - Skip producer-side batch coalescing when the input is unbounded. - Preserve the existing coalescing behavior for bounded inputs and preserve-order execution. - Add a regression test with an unbounded source that emits one partial batch and then remains open. ## Are these changes tested? Yes. - `cargo fmt --all -- --check` - `cargo test -p datafusion-physical-plan` - `cargo clippy --all-targets --all-features -- -D warnings` - The extended workspace test command required by `AGENTS.md`, including all 503 sqllogictest files The regression test fails on the previous implementation because the partial batch is never emitted, and passes with this change. The existing bounded-input coalescing test also continues to pass. ## Are there any user-facing changes? Yes. Repartitioning an unbounded input can now emit partial batches promptly instead of waiting indefinitely to fill the configured batch size. This may expose smaller batches to downstream operators. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
