Dale Richardson created YUNIKORN-3379:
-----------------------------------------
Summary: Scheduler panics on a malformed PLACEHOLDER_REPLACED
release with no linked replacement allocation
Key: YUNIKORN-3379
URL: https://issues.apache.org/jira/browse/YUNIKORN-3379
Project: Apache YuniKorn
Issue Type: Bug
Components: core - scheduler
Reporter: Dale Richardson
The core dereferences a nil pointer and crashes if it receives an inbound
AllocationRelease with TerminationType == PLACEHOLDER_REPLACED whose target
allocation has no linked replacement. A single malformed release message from
the RM takes the whole scheduler process down.
h3. Mechanism
{{processAllocationRelease}} (partition.go:1467) routes on the release message's
termination type alone and does not verify the target is a placeholder:
{code}
if release.TerminationType == si.TerminationType_PLACEHOLDER_REPLACED {
if alloc := app.ReplaceAllocation(allocationKey); alloc != nil { released =
append(released, alloc) }
}
{code}
{{Application.ReplaceAllocation}} (application.go:1966) deliberately tolerates a
placeholder with no linked replacement -- it logs "Placeholder replaced without
replacement allocation" and returns the allocation anyway, with
GetRelease()==nil.
That allocation flows into {{replacePlaceholderOnNode}} (partition.go:1596):
{code}
confirmed := alloc.GetRelease() // nil
delta := resources.Sub(confirmed.GetAllocatedResource(), ...) // RLock on nil
receiver -> panic
{code}
{{GetAllocatedResource}} (allocation.go:344) takes the allocation lock on the
nil
receiver and the scheduler dies with a nil-pointer panic.
h3. Reproduction
Direct: register any allocation with no release link, deliver an
AllocationRelease
with TerminationType_PLACEHOLDER_REPLACED for its key. Reproduced against
master;
it crashes the process (a partition-accounting property test had to exclude this
op because it aborts the test binary rather than failing an invariant).
h3. Reachability (checked)
Not reachable via the current Kubernetes shim. Placeholder replacement is a
two-phase protocol: the core notifies the RM (context.go:148), the shim deletes
the placeholder pod and echoes the termination type back as an inbound release
(task.go:455 -> UpdateAllocation). In that normal flow the placeholder always
has
a valid replacement link, so no panic. The one race that could clear the link --
a node removal during an in-flight replacement -- does NOT reach the panic:
{{removeNodeAllocations}} removes the placeholder from the application map
(partition.go:769-770 then 788), so a late inbound release hits
ReplaceAllocation's
nil/duplicate guard (application.go:1961-1963) and is a no-op. The panic
therefore
requires a *malformed* release (PLACEHOLDER_REPLACED for an allocation with no
linked replacement) that a correct RM does not send.
h3. Impact
Defensive robustness: the core must not crash on an unexpected or malformed RM
release. Low likelihood with the current trusted k8shim; higher concern for
other
or future RMs, and for any code path that could clear a link without removing
the
placeholder.
h3. Fix sketch
Handle GetRelease()==nil in {{replacePlaceholderOnNode}} (treat as a plain
placeholder removal, mirroring ReplaceAllocation's app-side tolerance), and/or
verify the target is a placeholder before routing into the replacement path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
