andygrove commented on PR #2419: URL: https://github.com/apache/datafusion-ballista/pull/2419#issuecomment-5533211551
_LLM-assisted reply (Claude Code), reviewed by me before posting._ Good call — filed as #2422 with a design sketch. Short version of what I found while writing it up: the post-AQE plan is *already* in the log, just unlabelled. `graph_to_job_response` renders `graph.physical_plan()`, and for `AdaptiveExecutionGraph` that getter returns the live `planner.plan`, so `JobEnd.job.physical_plan` is the plan after the last replan — whereas `JobStart.physical_plan` is the pre-execution one. Same field name, two meanings, and nothing marking the job as adaptive, which is why the plan dumps in this PR only ever showed the pre-execution plan. So the proposal is mostly about making that explicit (`final_physical_plan` + an `adaptive` flag on `JobResponse`, which rides inside the existing `JobEnd.job` blob and so needs no event-log schema bump), plus an opt-in per-replan `JobPlanRevision` event for the actual AQE decision history — off by default, since a full plan render per replan is expensive with the file-group listings. Open questions are in the issue; question 2 in particular is one for you if you have a view: `replan_stages` re-optimizes the whole tree after every stage completion and `ExchangeExec::with_new_children` carries the resolved shuffle state across a rewrite, so in principle a subtree under an already-resolved exchange could be rewritten after that stage ran — which would make the "final plan" disagree with the frozen `stage_plan`. Is that reachable, or prevented somewhere I missed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
