amitvijapur opened a new pull request, #2477:
URL: https://github.com/apache/datafusion-ballista/pull/2477
# Which issue does this PR close?
Closes #2418.
# Rationale for this change
Under `PushStaged`, the vcores held by running tasks are refunded in one
place: the `TaskUpdating` handler sums `task_vcores` for the statuses an
executor reports and calls `unbind_tasks`. `abort_job` cancels a job's running
tasks on the executors and then evicts the job from the active cache, so by the
time the executor reports those tasks as finished, `update_task_statuses` logs
`Fail to find job ... in the active cache` and drops the batch. That is the
third line in the issue's log, and it is why the refund never happens: every
cancelled task's vcores stay deducted from the executor's budget until the
scheduler has nothing left to bind.
# What changes are included in this PR?
`abort_job` now computes the vcores its running tasks hold, per executor,
from the graph snapshot taken right after `abort_running` (a failed stage keeps
its `task_infos`, so `task_vcores` still resolves), and passes them to the
cancel closure as a second argument. The scheduler's wrapper refunds them with
`unbind_tasks` once the executors have acknowledged the cancel, gated on
push-staged scheduling like the existing refund in `TaskUpdating`. If the
cancel RPC fails the refund is skipped. A lost executor has its whole vcore
budget dropped with it, so no refund applies there.
The same path handles `JobRunningFailed`, so a job that fails with tasks in
flight is covered too.
# Are these changes tested?
`aborted_tasks_are_cancelled_before_terminal_persistence` now also asserts
that the closure receives `[("executor-1", 1)]` for the one bound
single-partition task. `cargo test -p ballista-scheduler`: 424 passed, 2
ignored; `cargo clippy -p ballista-scheduler --all-targets -- -D warnings`
clean.
I did not rerun the issue's TPC-H reproduction end to end; the unit test
covers the refund and the log lines in the issue match the code path above.
# Are there any user-facing changes?
No API change. Cancelling or failing a running job under `PushStaged` no
longer shrinks the executor's schedulable vcores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
