avantgardnerio commented on PR #25798: URL: https://github.com/apache/datafusion/pull/25798#issuecomment-5936778377
Yes, I think Ballista would use it. Our range repartition operators (ORRE/URRE, apache/datafusion-ballista#2169, #2123) take their cuts from a RuntimeStatsExec sketch on the first batch they see. That's only a full sample because a SortExec between them acts as the barrier. Without one, the router sees a single batch. BufferExec (apache/datafusion-ballista#2095) works around this with memory-pool pressure, but that's a stand-in for exactly the prime > ready > inspect > release contract this PR defines. With StageBoundary we could repartition by range inside a stage without a sort or a pool-pressure heuristic. That's also the gap on the DataFusion side: RepartitionExec routes Partitioning::Range today, but its split points (and the samples from #24766) are fixed at plan time. Computing them from runtime data (#23093) needs a barrier like this one. I will give the PR a deeper review shortly, but this would allow partitioned windows and all the other use-cases I outlined in https://github.com/apache/datafusion/issues/23194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
