[
https://issues.apache.org/jira/browse/YUNIKORN-3448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116780#comment-18116780
]
Yen-Hua Chen commented on YUNIKORN-3448:
----------------------------------------
I'd like to work on this.
I reproduced the issue on current master and traced it to the
outstanding-request lifecycle: once an ask has advertised scale-up demand,
losing queue or user/group policy headroom removes it from the outstanding set
without withdrawing the previous advertisement. The existing
{{scaleUpTriggered}} state also prevents the ask from advertising again after
headroom is restored.
I have a local core-side fix and deterministic regressions covering the
{{FAILED -> SKIPPED -> FAILED}} lifecycle, duplicate suppression, accounting,
and the relevant policy-headroom boundaries. The focused scheduler/object tests
and race tests are passing locally after rebasing onto current master.
I'm doing an additional validation pass before preparing the PR. If there are
no objections, I'd like to take this issue.
> Revert autoscaling state when a scale-up-triggered ask loses queue headroom
> ---------------------------------------------------------------------------
>
> Key: YUNIKORN-3448
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3448
> Project: Apache YuniKorn
> Issue Type: Improvement
> Components: core - scheduler, shim - kubernetes
> Reporter: Kishan Ghetia
> Priority: Major
>
> A pod can become ineligible for scheduling after it has already triggered
> cluster autoscaling.
> Timeline:
> 1. A pod fits its queue and user headroom, but no existing node can
> accommodate it.
> 2. YuniKorn sends FAILED, the shim marks the pod Unschedulable, and Karpenter
> begins provisioning capacity.
> 3. YuniKorn sets scaleUpTriggered=true.
> 4. Before the new node becomes Ready, another allocation consumes the
> remaining queue, parent-queue, or user headroom.
> 5. The original pod no longer fits headroom, but YuniKorn does not reset
> scaleUpTriggered or change the pod’s Unschedulable status.
> 6. Karpenter can consequently provision a node that cannot be used by the
> triggering pod. The unused node may later be consolidated.
> The expected over-provisioning is likely small, but this can create
> unnecessary capacity and node churn.
>
> Proposed behavior:
> When a scale-up-triggered ask no longer fits queue or user headroom:
> - Reset its scaleUpTriggered state.
> - Send an appropriate update to the shim so the pod is no longer presented to
> the autoscaler as Unschedulable.
> - Allow the pod to trigger autoscaling again if headroom later becomes
> available and no suitable node exists.
>
> Acceptance criteria:
> - Cover the transition from autoscaling-eligible to headroom-ineligible.
> - Update both the core ask state and Kubernetes pod status.
> - Verify that the pod can trigger autoscaling again after regaining headroom.
> - Add a test where competing allocations consume headroom while capacity is
> being provisioned.
> - Preserve existing outstanding-request headroom accounting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
