[
https://issues.apache.org/jira/browse/YUNIKORN-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wilfred Spiegelenburg reassigned YUNIKORN-3375:
-----------------------------------------------
Assignee: Wilfred Spiegelenburg
> Node delete/re-add silently corrupts queue accounting: adopted pods are never
> re-reported to the core
> -----------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3375
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3375
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Assignee: Wilfred Spiegelenburg
> Priority: Major
>
> When a Node object is deleted and re-created while its pods keep running, the
> shim and the core end up permanently disagreeing about what is on that node.
> The shim has the pods placed; the core believes the node is empty and has
> given the queue its quota back. The queue's usage stays understated for as
> long as the pods run.
> *What works (this issue is not about these)*:
> - The orphaned pod tracking works. On node removal the pods are orphaned, and
> when the node comes back the adoption path puts them back on the node in the
> shim cache correctly.
> - There is no node-level double allocation. The shim's own predicates run
> against the shim cache, which contains the adopted pods, so a genuinely full
> node rejects new placements. We verified this.
> *The gap*: adoption only restores the shim's view. Nothing restores the
> core's.
> # On node removal the core removes the allocations from the applications and
> decrements the queue's allocated resources ({{removeNodeAllocations}} ->
> {{queue.DecAllocatedResource}}), then notifies the shim with
> {{TerminationType STOPPED_BY_RM}}. The shim callback correctly only forgets
> the pod in its cache and does not fail the task -- the pod is still running,
> and dispatching the release would delete it. The task stays Bound.
> # On node re-add, the node registration cannot carry allocations
> ({{existingAllocations}} was removed from the SI in
> [YUNIKORN-2780|https://issues.apache.org/jira/browse/YUNIKORN-2780]), and the
> adoption path never sends an allocation update either: {{updateYuniKornPod}}
> finds the still-Bound task and only calls {{SetTaskPod}}.
> # No reconciliation exists. The divergence lasts until each pod terminates,
> or until a scheduler restart (restart recovery does re-report).
> *Why we believe this is an oversight and not a design decision*: the same
> adoption loop DOES send an allocation update to the core for non-YuniKorn
> pods ({{updateForeignPod}} -> {{CreateAllocationForForeignPod}}, tracking
> added by
> [YUNIKORN-2834|https://issues.apache.org/jira/browse/YUNIKORN-2834]). Only
> the YuniKorn-managed branch is silent, because the surviving Bound task
> suppresses it.
> *Impact*:
> - Queue accounting is silently wrong: a queue with max=X can end up running
> more than X (the surviving pods plus a full X of new admissions elsewhere).
> User quotas, fair-share ordering, preemption inputs, autoscaler calculations,
> metrics and the web UI all compute against understated usage.
> - If ALL of an application's allocations were on the removed node, it is
> worse: the core application is left empty, moves Completing -> Completed
> (~30s default), and is removed from the partition entirely. The shim still
> holds the application with Bound tasks; the core no longer knows it exists.
> - The core also wastes scheduling cycles proposing a node it believes is
> empty while the shim rejects every attempt.
> *Reachability*: {{kubectl delete node}} on a live node (or automation doing
> the equivalent) followed by kubelet self-re-registration recreates the Node
> object within seconds. The pod-GC controller quarantines orphaned pods for
> ~40s before deleting them, so the pods surviving a brief flap is the normal
> outcome. Cluster-autoscaler is not a trigger (it drains first). Rare but
> ordinary operations; when it fires it affects every pod on the node.
> *Reproduction*: deterministic via the
> [YUNIKORN-3377|https://issues.apache.org/jira/browse/YUNIKORN-3377]
> fault-injection test ({{TestChaosNodeFlap}}, oracle O5): after a flap, e.g.
> one node shows 40 pods placed in the shim and zero allocations in the core,
> on every seed.
> *Suggested fix*: on adoption of a YuniKorn-managed pod whose task is Bound,
> re-report the allocation the way the foreign-pod branch and restart recovery
> already do ({{UpdateAllocation}} with the node ID set -> the core's
> {{MarkPreviouslyAllocated}} path). Because the core application may already
> have been removed (see impact above), the fix must follow the recovery
> sequence -- re-register the application first, then report the allocations --
> rather than sending the allocation update alone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
