andygrove commented on PR #11070: URL: https://github.com/apache/datafusion/pull/11070#issuecomment-2184166286
> Edit: Adding the distributed scheduler to this create would be great though and something we'd definitely be interested in using and contributing to, especially if if can abstract out the concrete implementation of actually shuffling data between stages I am also interested in this type of abstraction. I was thinking along the lines of having the planner insert `QueryStageExec` instead of `ShuffleWriterExec`. `QueryStageExec` would not have any execution logic. Responsibility for execution can be left to users of this planner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
