Dale Richardson created YUNIKORN-3412:
-----------------------------------------
Summary: Event system handler reads the fields Stop() rewrites;
Stop() blocks under the write lock
Key: YUNIKORN-3412
URL: https://issues.apache.org/jira/browse/YUNIKORN-3412
Project: Apache YuniKorn
Issue Type: Bug
Components: core - common
Reporter: Dale Richardson
{{EventSystemImpl.Stop()}} takes the write lock and then does two unbuffered
sends, {{ec.stop <- struct{}{}}} and the publisher's stop, before closing and
nil-ing {{ec.channel}}. The handler goroutine reads {{ec.stop}} and
{{ec.channel}} on every loop iteration without any lock.
Two problems share that seam. The blocking one: the publisher goroutine that
{{publisher.stop()}} waits for may be inside {{eventPlugin.SendEvent}}, a call
into the shim with no timeout. While it is, {{Stop()}} holds the write lock and
every {{AddEvent}} (read lock) queues behind it: the event system is dead until
the shim returns. The racing one: the handler's unlocked reads of the two
channel fields against {{Stop()}}'s writes. On master the second is masked by
the first (the blocking send orders the read, the receive and the write), and
fixing the send alone makes the race live; the two have to ship together. The
path is a config reload that toggles {{event.trackingEnabled}} ({{reloadConfig
-> restart -> Stop}}) and any in-process stop (YUNIKORN-3370). The wedge is
reproduced by two tests on the fork fix branch, which fail on master.
YUNIKORN-3336 and 3363 fixed the restart leak and the callback registration;
this is {{Stop()}} itself.
Fix: make {{Stop()}} CAS-first, snapshot and nil the fields under the lock,
keep {{close(ec.channel)}} inside the lock (because {{AddEvent}} sends under
the read lock), turn the two sends into closes after unlocking, and pass the
channels into the goroutines as parameters, the way {{CreateEventStream}}
already does. {{Stop()}} then no longer waits for the goroutines to
acknowledge. A fix exists on the tigerquoll fork
({{fix/event-system-stop-races}}, PR #15 there) and will be filed once this has
a number.
Marker: the handler goroutine in {{event_system.go}} carries this JIRA; the fix
removes it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
