metegenez commented on code in PR #1684:
URL:
https://github.com/apache/datafusion-ballista/pull/1684#discussion_r3253028757
##########
ballista/scheduler/src/state/aqe/planner.rs:
##########
@@ -296,8 +298,13 @@ impl AdaptivePlanner {
let (stage_ids, shuffle_writers) = stages
.into_iter()
.map(|plan| {
- // TODO: we need to find input stages for given stage
- // thus result should change
+ // Run the coalesce rule per-stage: the root of `plan`
is
+ // the stage's wrapper exchange, so the rule's walker
sees
+ // only this stage's input exchanges as the alignment
+ // group. This avoids cross-stage gluing and stale
state
+ // that would arise if the rule walked the entire
residual
+ // plan in `default_optimizers()`.
+ let plan = CoalescePartitionsRule.optimize(plan,
config)?;
Review Comment:
Inner stage optimisation will have its limits, for sure. Depending on the
follow up broadcast understanding, I can move this into a general rule. I
believe we will add more inner stage decisions at some point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
