[
https://issues.apache.org/jira/browse/YUNIKORN-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105185#comment-18105185
]
Wilfred Spiegelenburg commented on YUNIKORN-3374:
-------------------------------------------------
This change by itself without proper handling of the bind failures would leave
the system in a similar broken state. The cal to {{Assume()}} triggers an
update of the task state which also triggers the async bind of the pod to the
node. Without the proper handling of the volume and node binding errors that
should happen in this case the task will be left in an allocated state in the
k8shim and in a similar state on the core side.
This change cleans up the cache on the k8shim side it does not handle any of
the changes on the core side that should be triggered. For that we need
YUNIKORN-2884 as that reverts the core side and moves it back to scheduling via
the call to {{rescheduleOnBindFailure()}}
> Phantom allocation when a node with an assumed pod is removed and re-added
> --------------------------------------------------------------------------
>
> Key: YUNIKORN-3374
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3374
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Assignee: Dale Richardson
> Priority: Major
> Labels: pull-request-available
>
> When a node is removed while a pod is assumed on it (allocation made, bind
> not yet confirmed), the shim scheduler cache orphans the pod with the
> shim-invented {{Spec.NodeName}} still set. If the node re-appears (kubelet
> restart/re-registration, transient node object deletion and rejoin), the
> orphan adoption path re-assigns the never-bound pod to it and re-registers
> the task, and task recovery reports an existing allocation to the core for a
> pod that never ran: a phantom allocation that books queue and node capacity
> to nothing.
> Found by the invariant property fuzzer of YUNIKORN-3373. Sibling of
> YUNIKORN-3355 (same phantom-allocation class, different trigger: a node flap
> instead of a failed bind).
> *Mechanism* (all verified in code on master):
> # {{SchedulerCache.removeNode}} ({{scheduler_cache.go}} ~198) iterates the
> NodeInfo's pods and orphans ALL of them: it deletes the {{assignedPods}} and
> {{assumedPods}} entries and puts the pod into {{orphanedPods}}. The
> assumed/bound distinction is erased, and an assumed pod's cached object keeps
> the {{Spec.NodeName}} stamped at assume time -- a value the shim invented,
> which the apiserver's pod object never had.
> # {{updateNode}} on node (re-)add (~168) adopts any orphan whose
> {{Spec.NodeName}} matches the new node via {{updatePod}}: the phantom
> assignment is back.
> # {{Context.addNode}} ({{context.go}} ~211-218) registers each adopted pod
> via {{updateYuniKornPod}} -> {{ensureAppAndTaskCreated}}. The original task
> is typically gone by then (the core released the node's allocations when the
> node was removed), so {{addTask}} re-creates it with metadata carrying the
> stale-NodeName pod, and the recovery path reports the pod as an existing
> allocation on the re-added node.
> # The shim-cache half self-heals on the next informer update for the pod (it
> arrives unassigned), but the core-side allocation does not; it persists until
> released.
> Note the asymmetry that makes this a bug: for a genuinely BOUND pod,
> orphan-and-adopt is correct -- its node name is real and the pod actually
> runs on the flapping node. Only the assumed-never-bound case adopts a fiction.
> *Reachability*: requires a node delete + re-add around the assume-to-bind
> window of any pod. Node object flaps are routine at scale (kubelet
> re-registration, cloud node churn, node delete + rejoin), and the
> assume-to-bind window is always open somewhere on a busy cluster. In the
> YUNIKORN-3373 fuzzer, 50/50 seeds reach the state and observe the re-adoption
> (26,442 step observations over 250k steps; the fuzzer flaps nodes far more
> often than production, so this measures reachability, not frequency). A
> targeted regression test (assume, RemoveNode, UpdateNode, assert unassigned)
> fails on the unfixed code.
> *Fix* (PR to follow): in {{removeNode}}, treat assumed pods the way the
> YUNIKORN-3355 fix treats them in {{forgetPod}}: revert instead of orphan --
> drop the assumed/assigned state and re-store the pod unassigned (node name
> cleared on a copy). The pod is then simply rescheduled, which is the correct
> outcome as its target node is gone. Bound pods keep the existing orphan/adopt
> behaviour unchanged. The YUNIKORN-3373 fuzzer's invariants then guard the fix
> permanently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
