[
https://issues.apache.org/jira/browse/YUNIKORN-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YUNIKORN-3379:
-------------------------------------
Labels: pull-request-available (was: )
> Scheduler panics on a malformed PLACEHOLDER_REPLACED release with no linked
> replacement allocation
> ---------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3379
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3379
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Dale Richardson
> Assignee: Hedger Lai
> Priority: Minor
> Labels: pull-request-available
>
> The core dereferences a nil pointer and crashes if it receives an inbound
> AllocationRelease with TerminationType == PLACEHOLDER_REPLACED whose target
> allocation has no linked replacement. A single malformed release message from
> the RM takes the whole scheduler process down.
> h3. Mechanism
> {{processAllocationRelease}} (partition.go:1467) routes on the release
> message's
> termination type alone and does not verify the target is a placeholder:
> {code}
> if release.TerminationType == si.TerminationType_PLACEHOLDER_REPLACED {
> if alloc := app.ReplaceAllocation(allocationKey); alloc != nil { released
> = append(released, alloc) }
> }
> {code}
> {{Application.ReplaceAllocation}} (application.go:1966) deliberately
> tolerates a
> placeholder with no linked replacement -- it logs "Placeholder replaced
> without
> replacement allocation" and returns the allocation anyway, with
> GetRelease()==nil.
> That allocation flows into {{replacePlaceholderOnNode}} (partition.go:1596):
> {code}
> confirmed := alloc.GetRelease() // nil
> delta := resources.Sub(confirmed.GetAllocatedResource(), ...) // RLock on
> nil receiver -> panic
> {code}
> {{GetAllocatedResource}} (allocation.go:344) takes the allocation lock on the
> nil
> receiver and the scheduler dies with a nil-pointer panic.
> h3. Reproduction
> Direct: register any allocation with no release link, deliver an
> AllocationRelease
> with TerminationType_PLACEHOLDER_REPLACED for its key. Reproduced against
> master;
> it crashes the process (a partition-accounting property test had to exclude
> this
> op because it aborts the test binary rather than failing an invariant).
> h3. Reachability (checked)
> Not reachable via the current Kubernetes shim. Placeholder replacement is a
> two-phase protocol: the core notifies the RM (context.go:148), the shim
> deletes
> the placeholder pod and echoes the termination type back as an inbound release
> (task.go:455 -> UpdateAllocation). In that normal flow the placeholder always
> has
> a valid replacement link, so no panic. The one race that could clear the link
> --
> a node removal during an in-flight replacement -- does NOT reach the panic:
> {{removeNodeAllocations}} removes the placeholder from the application map
> (partition.go:769-770 then 788), so a late inbound release hits
> ReplaceAllocation's
> nil/duplicate guard (application.go:1961-1963) and is a no-op. The panic
> therefore
> requires a *malformed* release (PLACEHOLDER_REPLACED for an allocation with no
> linked replacement) that a correct RM does not send.
> h3. Impact
> Defensive robustness: the core must not crash on an unexpected or malformed RM
> release. Low likelihood with the current trusted k8shim; higher concern for
> other
> or future RMs, and for any code path that could clear a link without removing
> the
> placeholder.
> h3. Fix sketch
> Handle GetRelease()==nil in {{replacePlaceholderOnNode}} (treat as a plain
> placeholder removal, mirroring ReplaceAllocation's app-side tolerance), and/or
> verify the target is a placeholder before routing into the replacement path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
