[ https://issues.apache.org/jira/browse/YUNIKORN-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840044#comment-17840044 ]
Yu-Lin Chen commented on YUNIKORN-2576: --------------------------------------- [~wilfreds] Thanks for the review. You're right, the assertion fails is prior to the RACE issue. Let me create another Jira to clear asyncDispatchCount in initDispatcher(). (Also change to use atomic.Int32) I just did a quick test. After I added "asyncDispatchCount.Store(0)" to initDispatcher(), the RACE issue is temporarily gone. But I believet that the issue still exists. The DATA RACE log I put in this Jira shows that the race issue occurs in below code during unit test: * [https://github.com/chenyulin0719/yunikorn-k8shim/blob/64b204a2fb3b83fde9d86ea58f5f0d1e42187472/pkg/dispatcher/dispatcher.go#L73C1-L73C59] * [https://github.com/chenyulin0719/yunikorn-k8shim/blob/64b204a2fb3b83fde9d86ea58f5f0d1e42187472/pkg/dispatcher/dispatcher.go#L188C8-L188C18] Let's fix the assert issue first. > Data Race: Flaky tests in dispatcher_test.go > -------------------------------------------- > > Key: YUNIKORN-2576 > URL: https://issues.apache.org/jira/browse/YUNIKORN-2576 > Project: Apache YuniKorn > Issue Type: Bug > Components: test - unit > Reporter: Yu-Lin Chen > Priority: Major > Attachments: shim-race.txt > > > How to reproduce: > # In Shim, run 'go test ./pkg/... -race -count=10 > shim-race.txt' > > {code:java} > WARNING: DATA RACE > Write at 0x0000035315e0 by goroutine 88: > github.com/apache/yunikorn-k8shim/pkg/dispatcher.initDispatcher() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatcher.go:73 > +0x2c4 > github.com/apache/yunikorn-k8shim/pkg/dispatcher.createDispatcher() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatch_test.go:305 > +0x2f > runtime.Goexit() > /usr/local/go/src/runtime/panic.go:626 +0x5d > testing.(*T).FailNow() > <autogenerated>:1 +0x31 > gotest.tools/v3/assert.Equal() > > /home/chenyulin0719/go/pkg/mod/gotest.tools/v3@v3.5.1/assert/assert.go:205 > +0x1aa > github.com/apache/yunikorn-k8shim/pkg/dispatcher.TestDispatchTimeout() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatch_test.go:244 > +0x2ba > testing.tRunner() > /usr/local/go/src/testing/testing.go:1689 +0x21e > testing.(*T).Run.gowrap1() > /usr/local/go/src/testing/testing.go:1742 +0x44 > Previous read at 0x0000035315e0 by goroutine 90: > > github.com/apache/yunikorn-k8shim/pkg/dispatcher.(*Dispatcher).asyncDispatch.func1() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatcher.go:188 > +0x2f5 > > github.com/apache/yunikorn-k8shim/pkg/dispatcher.(*Dispatcher).asyncDispatch.gowrap1() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatcher.go:197 > +0x6eGoroutine 88 (running) created at: > testing.(*T).Run() > /usr/local/go/src/testing/testing.go:1742 +0x825 > testing.runTests.func1() > /usr/local/go/src/testing/testing.go:2161 +0x85 > testing.tRunner() > /usr/local/go/src/testing/testing.go:1689 +0x21e > testing.runTests() > /usr/local/go/src/testing/testing.go:2159 +0x8be > testing.(*M).Run() > /usr/local/go/src/testing/testing.go:2027 +0xf17 > main.main() > _testmain.go:55 +0x2bdGoroutine 90 (running) created at: > > github.com/apache/yunikorn-k8shim/pkg/dispatcher.(*Dispatcher).asyncDispatch() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatcher.go:178 > +0x391 > github.com/apache/yunikorn-k8shim/pkg/dispatcher.(*Dispatcher).dispatch() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatcher.go:164 > +0xbb > github.com/apache/yunikorn-k8shim/pkg/dispatcher.Dispatch() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatcher.go:142 > +0x71 > github.com/apache/yunikorn-k8shim/pkg/dispatcher.TestDispatchTimeout() > > /home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/dispatcher/dispatch_test.go:232 > +0x244 > testing.tRunner() > /usr/local/go/src/testing/testing.go:1689 +0x21e > testing.(*T).Run.gowrap1() > /usr/local/go/src/testing/testing.go:1742 +0x44 > =================={code} > Root Cause: > * The [globla > vairables|https://github.com/chenyulin0719/yunikorn-k8shim/blob/64b204a2fb3b83fde9d86ea58f5f0d1e42187472/pkg/dispatcher/dispatcher.go#L46-L51] > in dispatcher.go is not protected when running unit tests. Each unit test > will run initDispatcher() through > [createDispatcher()|https://github.com/chenyulin0719/yunikorn-k8shim/blob/64b204a2fb3b83fde9d86ea58f5f0d1e42187472/pkg/dispatcher/dispatch_test.go#L305]. > * Race occurs if any other unit tests read/write the global variables before > or after initDispatcher(). ex: TestDispatchTimeout() > [https://github.com/chenyulin0719/yunikorn-k8shim/blob/64b204a2fb3b83fde9d86ea58f5f0d1e42187472/pkg/dispatcher/dispatcher.go#L188] > > Solution to be discussed: > # Refactor dispatcher.go and encapsulates global variables to Dispatcher > struct > , change Dispatcher.Start(), Dispatcher.Stop() to type method > # Implement Singleton in getDispatcher() and add a new function > newDispatcher() > # Create a new Dispatcher for each unit test > > The race issue only happens in unit test becasue the shared vairable was > protected by > once.Do(initDispatcher) in dispatcher.go : > [https://github.com/chenyulin0719/yunikorn-k8shim/blob/64b204a2fb3b83fde9d86ea58f5f0d1e42187472/pkg/dispatcher/dispatcher.go#L131] > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@yunikorn.apache.org For additional commands, e-mail: issues-h...@yunikorn.apache.org