geoffreyclaude commented on PR #26242: URL: https://github.com/apache/datafusion/pull/26242#issuecomment-6340207237
> Thanks for addressing the earlier points. A few design questions after a second pass. @comphead Thanks for your very pertinent suggestions! I tested both on my fork, then investigated where the scheduling gains came from: - [[#22: coalesce small batches](https://github.com/geoffreyclaude/datafusion/pull/22)](https://github.com/geoffreyclaude/datafusion/pull/22). Fragmented-input sorts improved by 19 to 79%. This looks worth pursuing separately. Large inputs with normal-sized batches still need the task bound here. - [[#21: fixed groups of runs](https://github.com/geoffreyclaude/datafusion/pull/21)](https://github.com/geoffreyclaude/datafusion/pull/21). Unrelated query delays improved substantially, but wide and uneven-work sorts became 6 to 13% slower. It simplifies result collection while adding group balancing and more involved reservation ownership. I’d prefer keeping the current per-run design. - [[#23: yield during merge row-key initialization](https://github.com/geoffreyclaude/datafusion/pull/23)](https://github.com/geoffreyclaude/datafusion/pull/23). Traces pointed to merge initialization occupying all workers. Yielding directly there preserved the responsiveness gains without grouping runs: on wide input with 8,192-row batches, the median worst probe delay per run fell from 49.6 to 4.5 ms, with sort time changing by +0.7% (within noise). My proposed path is to keep bounded per-run preparation in this PR, then pursue coalescing and cooperative merge initialization as separate improvements in two new follow-up PRs. I also corrected the ordering comments and description: restoring run order keeps output independent of task completion order. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
