[
https://issues.apache.org/jira/browse/YUNIKORN-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3365:
--------------------------------------
Description:
{{ServiceContext.StopAll}} stops the scheduler before the RM proxy.
{{Scheduler.handleAllocEvent}} can select a queued allocation event over the
just-closed stop channel; the chain {{handleAllocEvent ->
handleRMUpdateAllocationEvent -> processAllocations -> notifyRMNewAllocation}}
then posts to the RM proxy and blocks on an unbuffered reply channel
({{context.go:803}}), but {{rmproxy.handleRMEvents}} returns on its own stop
without draining {{pendingRMEvents}}, so the reply never arrives and the
goroutine parks.
Reproduced in ~43% of full {{pkg/scheduler/tests}} runs (6/14): {{go test -race
-count=14 ./pkg/scheduler/tests/}}.
Severity is limited under the current lifecycle. As noted on PR #1124,
{{StopAll}} today runs as the process exits — the reply is moot and the queued
events are dropped anyway — so the parked goroutine is harmless in normal
operation. It matters only if (a) the parked goroutine blocks {{StopAll}} from
completing cleanly, or (b) the scheduler is stopped without the process
exiting, i.e. under in-process restart (YUNIKORN-3370).
{{notifyRMAllocationReleased}} has the same unbuffered-reply shape.
Proposed fix: drain or explicitly abandon in-flight replies on shutdown (or
invert the stop order), then delete the {{notifyRMNewAllocation}} exemption in
{{pkg/common/leakcheck/leakcheck.go}}.
was:
Follow-up to YUNIKORN-3357 (PR #1124); burns down the leakcheck exemption
scheduler.(*ClusterContext).notifyRMNewAllocation.
{{ServiceContext.StopAll}} stops the scheduler before the RM proxy.
{{Scheduler.handleAllocEvent}} can select a queued allocation event over the
just-closed stop channel; the chain {{handleAllocEvent ->
handleRMUpdateAllocationEvent -> processAllocations -> notifyRMNewAllocation}}
then posts to the RM proxy and blocks on an unbuffered reply channel
({{{}context.go:803{}}}), but {{rmproxy.handleRMEvents}} returns on its own
stop without draining {{{}pendingRMEvents{}}}, so the reply never arrives and
the goroutine is stuck permanently.
Reproduced in ~43% of full {{pkg/scheduler/tests}} runs (6/14):
{code:java}
go test -race -count=14 ./pkg/scheduler/tests/
{code}
{{notifyRMAllocationReleased}} has the same unbuffered-reply shape and would
leak under a different top frame; it has not been observed yet, so it is
deliberately not exempted — but the fix should cover it too.
Proposed fix: drain or explicitly abandon in-flight replies on shutdown (or
invert the stop order), then delete the exemption in
{{{}pkg/common/leakcheck/leakcheck.go{}}}. Argue both the deadlock and
data-integrity directions preemptively (cf. the YUNIKORN-2629 -> YUNIKORN-2910
regression).
> Shutdown ordering: scheduler allocation notify can block on an RM reply the
> proxy never drains
> ----------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3365
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3365
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Dale Richardson
> Priority: Major
> Labels: newbie
>
> {{ServiceContext.StopAll}} stops the scheduler before the RM proxy.
> {{Scheduler.handleAllocEvent}} can select a queued allocation event over the
> just-closed stop channel; the chain {{handleAllocEvent ->
> handleRMUpdateAllocationEvent -> processAllocations ->
> notifyRMNewAllocation}} then posts to the RM proxy and blocks on an
> unbuffered reply channel ({{context.go:803}}), but {{rmproxy.handleRMEvents}}
> returns on its own stop without draining {{pendingRMEvents}}, so the reply
> never arrives and the goroutine parks.
> Reproduced in ~43% of full {{pkg/scheduler/tests}} runs (6/14): {{go test
> -race -count=14 ./pkg/scheduler/tests/}}.
> Severity is limited under the current lifecycle. As noted on PR #1124,
> {{StopAll}} today runs as the process exits — the reply is moot and the
> queued events are dropped anyway — so the parked goroutine is harmless in
> normal operation. It matters only if (a) the parked goroutine blocks
> {{StopAll}} from completing cleanly, or (b) the scheduler is stopped without
> the process exiting, i.e. under in-process restart (YUNIKORN-3370).
> {{notifyRMAllocationReleased}} has the same unbuffered-reply shape.
> Proposed fix: drain or explicitly abandon in-flight replies on shutdown (or
> invert the stop order), then delete the {{notifyRMNewAllocation}} exemption
> in {{pkg/common/leakcheck/leakcheck.go}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
