andygrove commented on code in PR #2749:
URL:
https://github.com/apache/datafusion-ballista/pull/2749#discussion_r4642867898
##########
ballista/scheduler/src/state/aqe/mod.rs:
##########
@@ -657,6 +678,24 @@ impl AdaptiveExecutionGraph {
}
});
+ if let (Some(id), Some(registry)) = (running_stage_id, &mut
self.runtime_filters)
+ {
+ let plan = self.stages[&id].plan_arc();
+ let prepared = match registry.prepare(id, plan.clone(),
&self.stages) {
+ Ok(plan) => plan,
+ Err(error) => {
+ log::warn!(
+ "Runtime filter unavailable for stage {id}; bypassing:
{error}"
+ );
+ plan
+ }
+ };
+ let ExecutionStage::Running(stage) =
self.stages.get_mut(&id).unwrap() else {
+ unreachable!()
+ };
+ stage.plan = prepared;
Review Comment:
Thanks for the new end-to-end test, it caught a real interaction with #2875
that landed on `main`. That change caches `RunningStage::plan_metrics_len` when
the stage is created. Here we swap in the prepared plan afterwards, and when a
filter is bypassed that plan has one fewer operator. Every task's metrics then
get rejected as a shape mismatch, the stage succeeds with no metrics, and
`EXPLAIN ANALYZE` fails with `Missing metrics for operator
SortShuffleWriterExec`. It passes on the branch alone and fails once merged
with `main`.
Could you rebase and either refresh `plan_metrics_len` when the plan is
replaced, ideally through a small `RunningStage::replace_plan` so the two can't
drift again, or leave the unbound `RuntimeFilterApplyExec` in place as a
pass-through so the stage keeps its shape? I tried the first option locally and
the test passes.
##########
ballista/scheduler/src/state/aqe/mod.rs:
##########
@@ -1635,13 +1674,8 @@ impl ExecutionGraph for AdaptiveExecutionGraph {
let job_id = self.job_id.clone();
let session_id = self.session_id.clone();
- let mut next_task = self.stages.iter_mut().find(|(_stage_id, stage)| {
- if let ExecutionStage::Running(stage) = stage {
- stage.available_tasks() > 0
- } else {
- false
- }
- }).map(|(stage_id, stage)| {
+ let ready_id = self.get_running_stage_id(&[]);
Review Comment:
Small question here. Switching `pop_next_task` to `get_running_stage_id`
means it now also calls `revive()`, for every AQE job whether or not runtime
filters are on. Is that intended? If it's only there so `prepare` runs, could
it be limited to when `runtime_filters` is `Some`, or called out in the
description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]