geoffreyclaude opened a new pull request, #26931: URL: https://github.com/apache/datafusion/pull/26931
## Which issue does this PR close? Follow-up to #26242. Original experiment: [geoffreyclaude/datafusion#22](https://github.com/geoffreyclaude/datafusion/pull/22). ## Rationale for this change Small input batches create many tiny sort runs, increasing sorting-task and merge overhead. Combining adjacent batches reduces that overhead while retaining the shared task bound. ## What changes are included in this PR? Extend the existing coalescing helper to reach the configured `batch_size`, except for the final run, for all sort keys and before spilling. Preserve larger byte-sized coalescing for single-column final output. Entirely full-sized input passes through without concatenation. ## What is the testing strategy for this PR? Two focused tests cover row preservation, avoiding copies of standalone full batches, and multi-column StringView spilling with reservation cleanup. Keep the existing spill-reservation fixture testing separate runs. Formatting, full-workspace Clippy and extended workspace tests passed on the rebased branch. The October 22 fork experiment used SF1 lineitem on an M1 Max with four Tokio workers: narrow 32-row input sort time fell 79%, while ordinary preloaded wide input was near-neutral. These measurements use the earlier #26242 baseline; the linked fork PR contains the complete setup and results. ## Are there any user-facing changes? No new configuration or API. Equal-key rows may change relative order when batches become one run. Concatenation temporarily retains both input and output arrays. Prepared with AI assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
