Dale Richardson created YUNIKORN-3376:
-----------------------------------------
Summary: Data race on Task.pod: checkPodMetadataBeforeScheduling
reads without the task lock
Key: YUNIKORN-3376
URL: https://issues.apache.org/jira/browse/YUNIKORN-3376
Project: Apache YuniKorn
Issue Type: Bug
Components: shim - kubernetes
Reporter: Dale Richardson
{{Task.checkPodMetadataBeforeScheduling}} ({{pkg/cache/task.go}} ~523) reads
{{task.pod}} several times without holding the task lock. It runs on the shim
scheduling goroutine ({{Application.Schedule}} -> {{scheduleTasks}},
{{application.go}} ~403), concurrently with {{SetTaskPod}} ({{task.go}} ~676),
which writes the same field under {{task.lock}} from the informer handler
({{Context.updateYuniKornPod}}). The neighbouring {{checkPodPVCs}} takes
{{RLock}} for exactly the same field, so this is a missed lock, not a lock-free
design choice.
Confirmed by the race detector under load (read: {{scheduleTasks}} via the
{{wait.BackoffUntil}} scheduling loop; previous write: {{SetTaskPod}} via
{{Context.UpdatePod}} -- both production paths). An audit of every other
{{task.pod}} access in the file found them all correctly protected (directly or
inside FSM callbacks running under {{task.handle}}'s write lock), so this is
the only gap.
Impact is limited: the function is diagnostic-only (a warning log and a k8s
event), both racing values are complete immutable informer objects, and an
aligned pointer load does not tear on supported platforms. Filed because it is
formal Go-memory-model UB, it can flake any {{-race}} CI run, and the fix is
one line: snapshot {{pod := task.GetTaskPod()}} at the top of the function and
use it (including in {{logIgnoredPodMetadata}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
