[
https://issues.apache.org/jira/browse/YUNIKORN-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113106#comment-18113106
]
Yen-Hua Chen commented on YUNIKORN-3382:
----------------------------------------
I saw that someone has just asked to take this issue. I don’t want to duplicate
work unnecessarily, but I’ve already been investigating YUNIKORN-3382
independently and have some concrete results that may be useful.
I reproduced the missing re-drive deterministically with a focused shim-level
regression. I also built a local prototype where a failed victim Pod delete is
re-driven asynchronously after the original application release handler
returns. In the focused test, this changes the regression from RED to GREEN and
reconnects to the existing informer-driven completion path.
While testing the retry path, I also found a potential identity-safety issue
with name-only retries: a stale retry could act on a same-name replacement Pod.
The prototype uses a Kubernetes UID precondition, and the corresponding test
keeps the replacement Pod intact.
The prototype is still experimental rather than upstream-ready. The remaining
questions are mainly retry lifetime/exhaustion, duplicate outstanding retries,
lifecycle cancellation, and error classification.
I’d be happy to work on the production fix or coordinate on it if that is
preferable. My current direction is a small shim-side asynchronous re-drive
while preserving the existing informer-driven completion semantics.
> Failed victim pod delete permanently wedges a preemption; release-path
> DeleteTaskPod is never retried
> -----------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3382
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3382
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler, shim - kubernetes
> Reporter: Dale Richardson
> Priority: Major
>
> When the shim executes a core-initiated release by deleting the pod, a failed
> delete is logged and dropped -- there is no retry on this path. One transient
> apiserver failure leaves the release permanently half-executed.
> h3. Mechanism
> The core marks a preemption victim and hands it to the RM to release, but
> does not
> free its capacity until the shim confirms:
> - victim.MarkPreempted() at preempt time (preemption.go:659), before the RM is
> notified -- so the victim is excluded from all future victim selection
> (queue.go:2206) from that moment;
> - victimQueue.IncPreemptingResource(...) (preemption.go:681);
> - p.ask.MarkTriggeredPreemption() (preemption.go:702);
> - notifyRMAllocationReleased(PREEMPTED_BY_SCHEDULER) (preemption.go:705).
> The victim's node/queue release and DecPreemptingResource run only when the
> shim's
> PREEMPTED_BY_SCHEDULER confirmation returns (partition.go:1562-1563, gated by
> the
> "this release is the shim's confirmation" logic at partition.go:1577-1579).
> The shim executes the release by deleting the pod, and drops any failure:
> {code}
> // Application.handleReleaseAppAllocationEvent (application.go:648-655)
> err := task.DeleteTaskPod()
> if err != nil {
> log.Log(...).Error("failed to release allocation from application",
> zap.Error(err))
> }
> {code}
> Nothing re-queues the task, no timer retries the delete, and the core is never
> told the release did not happen. MarkUnPreempted only runs on the "victim
> already
> released" revert (preemption.go:669), never on the failed-delete path. The
> result
> is permanent:
> - the victim keeps running and stays IsPreempted forever, excluded from future
> preemption;
> - the victim queue's preemptingResource is never decremented, so it
> under-reports
> its preemptable and remaining-guaranteed headroom by the victim's size;
> - the high-priority ask the preemption was made for was marked as having
> triggered
> preemption and now starves -- its capacity never arrives.
> h3. Reproduction
> Chaos scenario: two-queue guaranteed/max config, cluster filled with
> low-priority
> pods, high-priority wave forces preemption, victim deletes failed with p=0.5.
> After settling, the victim runs on both sides consistently (no accounting
> divergence), the queue holds a non-zero preemptingResource forever, and one
> high-priority pod is never placed. One failed delete per wedge; deterministic
> per
> seed. The same dropped-delete leaves released gang placeholders alive
> indefinitely.
> h3. Impact
> Transient apiserver delete failures (timeouts, webhook hiccups) are routine in
> busy clusters. Each occurrence silently degrades preemption capacity and
> guaranteed-quota enforcement for the victim queue until a scheduler restart,
> and
> starves the ask that triggered the preemption.
> h3. Fix sketch
> Retry the delete (bounded backoff) or re-drive it from a reconciliation pass,
> or
> report the failed release back to the core so it can re-issue. The single
> release
> path above covers both the preemption and the gang-placeholder consequence.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
