[
https://issues.apache.org/jira/browse/YUNIKORN-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111279#comment-18111279
]
Hedger Lai commented on YUNIKORN-3380:
--------------------------------------
PR created: https://github.com/apache/yunikorn-core/pull/1147. Ready for
review. Thanks!
> RemoveAllAllocations leaks user/group tracked usage when a release-all
> arrives with pending asks
> ------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3380
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3380
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Dale Richardson
> Assignee: Hedger Lai
> Priority: Minor
> Labels: pull-request-available
>
> A release-all (AllocationRelease with an empty allocation key) that is
> processed
> while the application still has a pending ask permanently leaks the
> application's
> allocated resources into the user/group tracker. The leaked amount reduces the
> user's headroom until the scheduler restarts.
> h3. Mechanism
> {{Application.RemoveAllAllocations}} (application.go:2111):
> {code}
> if ugm.GetUserManager().GetUserTracker(sa.user.User) != nil &&
> resources.IsZero(sa.pending) {
> sa.decUserResourceUsage(resources.Add(sa.allocatedResource,
> sa.allocatedPlaceholder), true)
> }
> sa.allocatedResource = resources.NewResource() // wiped unconditionally
> sa.allocatedPlaceholder = resources.NewResource()
> {code}
> The tracker decrement is gated on IsZero(sa.pending), but the totals are wiped
> right after regardless. With pending non-zero the decrement is skipped and can
> never happen (the totals are already zero). On the release-all path,
> {{removeAllocation}} then drops the asks (partition.go:1585), so pending ends
> at
> zero and the state looks clean everywhere except inside the tracker.
> The app-removal path is NOT affected: {{removeApplication}} removes all asks
> first
> (partition.go:422), so pending is already zero when RemoveAllAllocations runs
> and
> the decrement fires.
> h3. Reproduction
> App with one running allocation and one pending ask; deliver an empty-key
> release-all; the user and group trackers permanently report the leaked usage
> with
> no owning application. Subsequent asks by that user are rejected with
> NotEnoughUserQuota once the leak eats the limit. {{TestRemoveAppAllocs}}
> exercises
> the same call only with no pending asks, which is why it is uncaught.
> Reproduced
> against master via a partition-accounting property test (its user/group
> invariant
> fires on this within seconds once release-all is generated).
> h3. Reachability (checked)
> Not emitted by the current Kubernetes shim. App removal uses the
> RemoveApplication
> request (the asks-first path, no leak); per-task releases always carry the
> pod UID
> as the allocation key (si_helper.go:154). {{removeAllocation}} is driven only
> by
> inbound RM releases (context.go:779). The empty-key release-all branch
> (partition.go:1461) is a deliberately-supported SI message, so the bug is
> latent
> in the core/SI contract and reachable by any RM that sends it (or by a future
> k8shim change), but not by the shim as it stands today.
> h3. Impact
> Silent, permanent per-occurrence quota corruption in clusters using user/group
> limits, cleared only by restart. Latent today; correctness bug in a supported
> SI
> code path.
> h3. Fix sketch
> Decrement the tracker guarded only by tracker existence (drop the
> IsZero(pending)
> condition) before the totals are wiped; the pending state of asks is
> irrelevant to
> returning resources that were tracked as allocated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
