andygrove opened a new pull request, #2149:
URL: https://github.com/apache/datafusion-ballista/pull/2149
# Which issue does this PR close?
Closes #1996.
# Rationale for this change
#1982 reclaims a job's intermediate shuffle data as soon as the job
succeeds, instead of waiting for `finished_job_data_clean_up_interval_seconds`
(default 300s). It picks the stages to reclaim with
`ExecutionGraph::intermediate_stage_ids()`, which selects stages whose
`output_links` is non-empty.
That works for `StaticExecutionGraph`, where `ExecutionStageBuilder` fills
in `output_links` up front. It does not work for `AdaptiveExecutionGraph`. The
adaptive planner creates stages incrementally in `create_resolved_stage` and
always passes `vec![]` for `output_links` ("we do not know output links at this
moment"), and nothing ever back-fills it. So under AQE every stage looks final,
the intermediate set is empty, and the immediate reclaim is a no-op. The job
then falls back to the delayed whole-job cleanup, which for a short benchmark
run never fires before the disk fills up.
This is the AQE-on half of the SF10 CI job in #1981 getting no relief at
all. It showed up again on #2064, where query 21 failed with `No space left on
device (os error 28)` while the AQE-off leg passed.
# What changes are included in this PR?
Override `intermediate_stage_ids()` for `AdaptiveExecutionGraph` to derive
the final stages from `output_locations` rather than `output_links`. Each
`PartitionLocation` carries the terminal `stage_id` that produced it, so the
final set is the stage ids present in `output_locations` and everything else is
intermediate.
`output_locations` is only populated once the planner has no more stages to
run, so while the job is still running it is empty and the override returns an
empty set. That preserves the "never delete a final stage" invariant from #1982.
Adds an AQE test that drives a multi-stage adaptive job to success and
asserts that `intermediate_stage_ids()` returns exactly the non-terminal
stages, that it is disjoint from the stages in `output_locations`, and that it
is empty before the job produces output.
# Are there any user-facing changes?
No API changes. Jobs running under the adaptive planner now free their
intermediate shuffle files as soon as the job succeeds, matching the behavior
the static planner already had.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
