[
https://issues.apache.org/jira/browse/YUNIKORN-3449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikandan R resolved YUNIKORN-3449.
------------------------------------
Fix Version/s: 1.11.0
Target Version: 1.11.0
Resolution: Fixed
> [Core] Preemption falsely aborts and starves large asks by ignoring node
> available capacity in shortfall check
> --------------------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3449
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3449
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Affects Versions: 1.6.0
> Reporter: Hedger Lai
> Assignee: Hedger Lai
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> h3. Problem Summary
> In {{Preemptor.TryPreemption()}}, when selecting preemption victims on a
> candidate node, upstream {{tryNodes()}} and {{calculateVictimsByNode()}}
> evaluate victims by adding their resources to {{nodeAvailable}}:
> {code:go}
> nodeCurrentAvailable.AddTo(victim.GetAllocatedResource())
> if nodeCurrentAvailable.FitIn(p.ask.GetAllocatedResource()) && index < 0 {
> index = len(results)
> }
> {code}
> Upstream correctly stops collecting victims once {{nodeAvailable + victims >=
> ask}}. That is, upstream only selects victims to cover the node's residual
> *deficit* ({{ask - nodeAvailable}}).
> However, downstream in {{TryPreemption()}} (lines 670-678 on master), the
> shortfall validation compares {{victimsTotalResource}} directly against the
> full ask requirement ({{p.ask.GetAllocatedResource()}}):
> {code:go}
> hasShortfall := victimsTotalResource.IsEmpty()
> if !hasShortfall {
> for k, victimVal := range victimsTotalResource.Resources {
> if needVal, ok := p.ask.GetAllocatedResource().Resources[k]; ok &&
> victimVal < needVal {
> hasShortfall = true
> break
> }
> }
> }
> if hasShortfall {
> // there is shortfall, so preemption doesn't help
> p.ask.LogAllocationFailure(common.PreemptionShortfall, true)
> return nil, false
> }
> {code}
> Because {{victimsTotalResource}} only accumulates victim allocations without
> factoring in the node's existing available capacity
> ({{p.nodeAvailableMap[nodeID]}}), preemption is falsely aborted whenever a
> candidate node has partial available capacity!
> h3. Reproduction Scenario
> Consider a node with 4 vcores capacity:
> # *Node available capacity*: 2 vcores idle ({{nodeAvailable = 2}}).
> # *Running allocation*: Pod A using 2 vcores (candidate victim).
> # *Pending Ask*: Requires 4 vcores.
> * *Upstream behavior*:
> {{calculateVictimsByNode()}} starts with 2 free vcores. It selects Pod A (2
> vcores). {{nodeCurrentAvailable}} becomes {{2 + 2 = 4 vcores}}, which fits
> the ask. Upstream returns {{nodeVictims = [Pod A]}} (2 vcores).
> * *Downstream failure*:
> {{victimsTotalResource}} = 2 vcores.
> Downstream evaluates {{victimVal (2) < needVal (4)}} -> {{hasShortfall =
> true}}.
> {{TryPreemption()}} aborts with {{PreemptionShortfall}}, even though
> preempting Pod A would free up the 4 vcores needed to schedule the ask!
> h3. Root Cause
> In {{TryPreemption()}}, the shortfall check assumes that the total resources
> of all selected victims alone must equal or exceed the ask's resource
> requirement. It completely ignores that the selected node already contributes
> {{p.nodeAvailableMap[nodeID]}} towards fulfilling the ask.
> h3. Proposed Solution
> When checking for preemption shortfall on the chosen node:
> # Factor in the node's existing available capacity:
> The required physical capacity from victims on the node is {{deficit = max(0,
> ask - nodeAvailable)}}.
> # Ensure the shortfall check verifies whether victims satisfy the node
> deficit (for physical node capacity) as well as any queue headroom
> requirements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
