[
https://issues.apache.org/jira/browse/YUNIKORN-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Huang reassigned YUNIKORN-3422:
------------------------------------
Assignee: Andy Huang
> SchedulerCache node-list getters populate their cache under the read lock
> -------------------------------------------------------------------------
>
> Key: YUNIKORN-3422
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3422
> Project: Apache YuniKorn
> Issue Type: Sub-task
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Assignee: Andy Huang
> Priority: Major
>
> {{SchedulerCache.GetNodesInfo}}, {{GetNodesInfoPodsWithAffinity}} and
> {{GetNodesInfoPodsWithReqAntiAffinity}} lazily build their list and assign it
> to the cache field the first time they are called after an invalidation.
> Their only callers are the predicate shared lister, reached from
> {{IsPodFitNode}} / {{IsPodFitNodeViaPreemption}} under {{LockForReads}}, i.e.
> the read lock.
> {code}func (cache *SchedulerCache) GetNodesInfo() []fwk.NodeInfo {
> if cache.nodesInfo == nil {
> nodeList := make([]fwk.NodeInfo, 0, len(cache.nodesMap))
> [...]
> cache.nodesInfo = nodeList // written under the read lock
> }
> return cache.nodesInfo
> }
> {code}
> The core's preemption runs predicate checks on up to ten goroutines at once
> ({{preemptCheckConcurrency}}), so two of them regularly see {{nil}} together
> and both write. Confirmed with the race detector on that path during the July
> review (no test in the tree). Concurrent writes of a slice header can publish
> a torn one, which would panic inside PreFilter; that consequence is traced,
> not observed. Invalidation ({{nodesInfo = nil}}) is already under the write
> lock; only the population is wrong.
> Fix: build the three lists at invalidation time, under the write lock already
> held there, or publish them through an {{atomic.Pointer}}. Fix all three
> getters together.
> Marker: three {{+checklocksignore}} sites in {{scheduler_cache.go}} carry
> this JIRA; the fix removes them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
