Dale Richardson created YUNIKORN-3380:
-----------------------------------------
Summary: RemoveAllAllocations leaks user/group tracked usage when
a release-all arrives with pending asks
Key: YUNIKORN-3380
URL: https://issues.apache.org/jira/browse/YUNIKORN-3380
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Reporter: Dale Richardson
A release-all (AllocationRelease with an empty allocation key) that is processed
while the application still has a pending ask permanently leaks the
application's
allocated resources into the user/group tracker. The leaked amount reduces the
user's headroom until the scheduler restarts.
h3. Mechanism
{{Application.RemoveAllAllocations}} (application.go:2111):
{code}
if ugm.GetUserManager().GetUserTracker(sa.user.User) != nil &&
resources.IsZero(sa.pending) {
sa.decUserResourceUsage(resources.Add(sa.allocatedResource,
sa.allocatedPlaceholder), true)
}
sa.allocatedResource = resources.NewResource() // wiped unconditionally
sa.allocatedPlaceholder = resources.NewResource()
{code}
The tracker decrement is gated on IsZero(sa.pending), but the totals are wiped
right after regardless. With pending non-zero the decrement is skipped and can
never happen (the totals are already zero). On the release-all path,
{{removeAllocation}} then drops the asks (partition.go:1585), so pending ends at
zero and the state looks clean everywhere except inside the tracker.
The app-removal path is NOT affected: {{removeApplication}} removes all asks
first
(partition.go:422), so pending is already zero when RemoveAllAllocations runs
and
the decrement fires.
h3. Reproduction
App with one running allocation and one pending ask; deliver an empty-key
release-all; the user and group trackers permanently report the leaked usage
with
no owning application. Subsequent asks by that user are rejected with
NotEnoughUserQuota once the leak eats the limit. {{TestRemoveAppAllocs}}
exercises
the same call only with no pending asks, which is why it is uncaught. Reproduced
against master via a partition-accounting property test (its user/group
invariant
fires on this within seconds once release-all is generated).
h3. Reachability (checked)
Not emitted by the current Kubernetes shim. App removal uses the
RemoveApplication
request (the asks-first path, no leak); per-task releases always carry the pod
UID
as the allocation key (si_helper.go:154). {{removeAllocation}} is driven only by
inbound RM releases (context.go:779). The empty-key release-all branch
(partition.go:1461) is a deliberately-supported SI message, so the bug is latent
in the core/SI contract and reachable by any RM that sends it (or by a future
k8shim change), but not by the shim as it stands today.
h3. Impact
Silent, permanent per-occurrence quota corruption in clusters using user/group
limits, cleared only by restart. Latent today; correctness bug in a supported SI
code path.
h3. Fix sketch
Decrement the tracker guarded only by tracker existence (drop the
IsZero(pending)
condition) before the totals are wiped; the pending state of asks is irrelevant
to
returning resources that were tracked as allocated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
