[
https://issues.apache.org/jira/browse/YUNIKORN-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113112#comment-18113112
]
Dian-Xuan Yang commented on YUNIKORN-3231:
------------------------------------------
I investigated this and was able to reproduce the behavior on 1.8.0.
It does not appear to be related to `nodeSelector` or `nodeAffinity`. The
actual issue is the informer event order, which matches the behavior fixed by
YUNIKORN-3317 in 1.9.0.
I suggest closing YUNIKORN-3231 as a duplicate of YUNIKORN-3317.
> ForeignAllocations ignored / not accounted when pods use nodeSelector (works
> with nodeAffinity)
> -----------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3231
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3231
> Project: Apache YuniKorn
> Issue Type: Bug
> Environment: Kubernetes: 1.26
> YuniKorn scheduler: 1.8.0
> Reporter: Alex Ovchenkov
> Priority: Major
>
> h2. Description
> We observed that YuniKorn’s *ForeignAllocations* (pods not managed by
> YuniKorn / allocations coming from outside) may not be accounted correctly
> when such pods use {*}nodeSelector{*}. As a result, YuniKorn may overestimate
> node/queue available resources and make scheduling decisions as if those
> external pods do not exist (or are not tied to the expected nodes).
> When we replace {{nodeSelector}} with an equivalent
> {{{}nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution{}}}, the
> behavior becomes correct and ForeignAllocations are accounted as expected.
> h2. *Steps to reproduce*
> h5. 1) Create a pod that is scheduled by Kubernetes default scheduler (or
> another scheduler) and uses nodeSelector:
>
> {code:java}
> apiVersion: v1
> kind: Pod
> metadata:
> name: foreign-pod-selector
> namespace: <ns>
> spec:
> schedulerName: default-scheduler
> nodeSelector:
> dedicated: test
> containers:
> - name: c
> image: busybox
> command: ["sh", "-c", "sleep 360000"]
> resources:
> requests:
> cpu: "1"
> memory: "1Gi" {code}
> h5. 2) Observe YuniKorn resource accounting / foreign allocations
> Check YuniKorn UI / REST API / state dump and verify whether this pod appears
> as a foreign allocation and whether its resources are deducted from the node.
> h5. 3) Replace nodeSelector with equivalent nodeAffinity
> (requiredDuringScheduling)
> Delete the pod and apply the same pod but with nodeAffinity instead:
> {code:java}
> apiVersion: v1
> kind: Pod
> metadata:
> name: foreign-pod-affinity
> namespace: <ns>
> spec:
> schedulerName: default-scheduler
> affinity:
> nodeAffinity:
> requiredDuringSchedulingIgnoredDuringExecution:
> nodeSelectorTerms:
> - matchExpressions:
> - key: dedicated
> operator: In
> values: ["test"]
> containers:
> - name: c
> image: busybox
> command: ["sh", "-c", "sleep 360000"]
> resources:
> requests:
> cpu: "1"
> memory: "1Gi" {code}
> Re-check YuniKorn accounting / foreign allocations.
> ----
> h2. Actual behavior
> * With {*}nodeSelector{*}, the foreign pod is *not consistently* represented
> in ForeignAllocations / node accounting (or appears without correct node
> association / resources not deducted).
> * This can lead to YuniKorn scheduling decisions that assume more resources
> are available on the affected node(s) than in reality.
> ----
> h2. Expected behavior
> * Foreign pods should be accounted consistently regardless of whether the
> workload uses:
> ** {{{}spec.nodeSelector{}}}, or
> **
> {{spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution}}
> * If the pod is bound to a node (and consuming resources), YuniKorn should
> reflect it in foreign allocations / node resource usage.
> ----
> h2. Notes / hypothesis
> Based on initial investigation, the issue might be related to how foreign
> pods are processed in {{{}updateForeignPod{}}}:
> [https://github.com/apache/yunikorn-k8shim/blob/master/pkg/cache/context.go#L412]
> It appears that when a pod uses {{{}nodeSelector{}}}, some of its
> scheduling-related state (possibly node assignment or allocation state) may
> already be set earlier in the reconciliation flow, which could cause
> {{updateForeignPod}} to skip or treat it differently.
> In contrast, when equivalent constraints are expressed using
> {{{}nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution{}}}, the
> foreign allocation seems to be tracked and accounted correctly.
> This suggests there may be a difference in how {{nodeSelector}} and
> {{nodeAffinity}} are interpreted or propagated into the internal foreign
> allocation tracking logic.
> We may be missing some detail in the lifecycle of foreign pod updates, but
> the observable behavior indicates inconsistent accounting depending on
> whether {{nodeSelector}} or {{nodeAffinity}} is used.
> ----
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
