[
https://issues.apache.org/jira/browse/YUNIKORN-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YUNIKORN-3374:
-------------------------------------
Labels: pull-request-available (was: )
> Phantom allocation when a node with an assumed pod is removed and re-added
> --------------------------------------------------------------------------
>
> Key: YUNIKORN-3374
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3374
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Priority: Major
> Labels: pull-request-available
>
> When a node is removed while a pod is assumed on it (allocation made, bind
> not yet confirmed), the shim scheduler cache orphans the pod with the
> shim-invented {{Spec.NodeName}} still set. If the node re-appears (kubelet
> restart/re-registration, transient node object deletion and rejoin), the
> orphan adoption path re-assigns the never-bound pod to it and re-registers
> the task, and task recovery reports an existing allocation to the core for a
> pod that never ran: a phantom allocation that books queue and node capacity
> to nothing.
> Found by the invariant property fuzzer of YUNIKORN-3373. Sibling of
> YUNIKORN-3355 (same phantom-allocation class, different trigger: a node flap
> instead of a failed bind).
> *Mechanism* (all verified in code on master):
> # {{SchedulerCache.removeNode}} ({{scheduler_cache.go}} ~198) iterates the
> NodeInfo's pods and orphans ALL of them: it deletes the {{assignedPods}} and
> {{assumedPods}} entries and puts the pod into {{orphanedPods}}. The
> assumed/bound distinction is erased, and an assumed pod's cached object keeps
> the {{Spec.NodeName}} stamped at assume time -- a value the shim invented,
> which the apiserver's pod object never had.
> # {{updateNode}} on node (re-)add (~168) adopts any orphan whose
> {{Spec.NodeName}} matches the new node via {{updatePod}}: the phantom
> assignment is back.
> # {{Context.addNode}} ({{context.go}} ~211-218) registers each adopted pod
> via {{updateYuniKornPod}} -> {{ensureAppAndTaskCreated}}. The original task
> is typically gone by then (the core released the node's allocations when the
> node was removed), so {{addTask}} re-creates it with metadata carrying the
> stale-NodeName pod, and the recovery path reports the pod as an existing
> allocation on the re-added node.
> # The shim-cache half self-heals on the next informer update for the pod (it
> arrives unassigned), but the core-side allocation does not; it persists until
> released.
> Note the asymmetry that makes this a bug: for a genuinely BOUND pod,
> orphan-and-adopt is correct -- its node name is real and the pod actually
> runs on the flapping node. Only the assumed-never-bound case adopts a fiction.
> *Reachability*: requires a node delete + re-add around the assume-to-bind
> window of any pod. Node object flaps are routine at scale (kubelet
> re-registration, cloud node churn, node delete + rejoin), and the
> assume-to-bind window is always open somewhere on a busy cluster. In the
> YUNIKORN-3373 fuzzer, 50/50 seeds reach the state and observe the re-adoption
> (26,442 step observations over 250k steps; the fuzzer flaps nodes far more
> often than production, so this measures reachability, not frequency). A
> targeted regression test (assume, RemoveNode, UpdateNode, assert unassigned)
> fails on the unfixed code.
> *Fix* (PR to follow): in {{removeNode}}, treat assumed pods the way the
> YUNIKORN-3355 fix treats them in {{forgetPod}}: revert instead of orphan --
> drop the assumed/assigned state and re-store the pod unassigned (node name
> cleared on a copy). The pod is then simply rescheduled, which is the correct
> outcome as its target node is gone. Bound pods keep the existing orphan/adopt
> behaviour unchanged. The YUNIKORN-3373 fuzzer's invariants then guard the fix
> permanently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
