Yu-Lin Chen created YUNIKORN-2559:
-------------------------------------
Summary: DATA RACE: EventStore.Store() and Context.PublishEvents()
Key: YUNIKORN-2559
URL: https://issues.apache.org/jira/browse/YUNIKORN-2559
Project: Apache YuniKorn
Issue Type: Bug
Components: core - common
Reporter: Yu-Lin Chen
Assignee: Yu-Lin Chen
Attachments: shim-racing-log.txt
How to reproduce:
# In shim, update core version to the latest version
(v0.0.0-20240415111844-72540e2b277f)
# go mod tidy
# Run 'make test > shim-racing-log.txt'
{code:java}
WARNING: DATA RACE
Write at 0x00c003882008 by goroutine 59:
github.com/apache/yunikorn-core/pkg/events.(*EventStore).Store()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_store.go:59
+0x1aa
github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher.func2()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:194
+0x167Previous read at 0x00c003882008 by goroutine 60:
github.com/apache/yunikorn-k8shim/pkg/cache.(*Context).PublishEvents()
/home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/cache/context.go:1176
+0x97
github.com/apache/yunikorn-k8shim/pkg/cache.(*AsyncRMCallback).SendEvent()
/home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/cache/scheduler_callback.go:235
+0xec
github.com/apache/yunikorn-core/pkg/events.(*EventPublisher).StartService.func1()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_publisher.go:60
+0x27dGoroutine 59 (running) created at:
github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:183
+0x287
github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartService()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:166
+0x2b
github.com/apache/yunikorn-core/pkg/entrypoint.startAllServicesWithParameters()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:80
+0x9b
github.com/apache/yunikorn-core/pkg/entrypoint.StartAllServices()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:43
+0x59
github.com/apache/yunikorn-k8shim/pkg/shim.(*MockScheduler).init()
/home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_mock_test.go:63
+0xad
github.com/apache/yunikorn-k8shim/pkg/shim.TestApplicationScheduling()
/home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_test.go:60 +0x8c
testing.tRunner()
/usr/local/go/src/testing/testing.go:1689 +0x21e
testing.(*T).Run.gowrap1()
/usr/local/go/src/testing/testing.go:1742 +0x44Goroutine 60 (running)
created at:
github.com/apache/yunikorn-core/pkg/events.(*EventPublisher).StartService()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_publisher.go:50
+0xc4
github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartServiceWithPublisher()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:203
+0x2b8
github.com/apache/yunikorn-core/pkg/events.(*EventSystemImpl).StartService()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/events/event_system.go:166
+0x2b
github.com/apache/yunikorn-core/pkg/entrypoint.startAllServicesWithParameters()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:80
+0x9b
github.com/apache/yunikorn-core/pkg/entrypoint.StartAllServices()
/home/chenyulin0719/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20240415111844-72540e2b277f/pkg/entrypoint/entrypoint.go:43
+0x59
github.com/apache/yunikorn-k8shim/pkg/shim.(*MockScheduler).init()
/home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_mock_test.go:63
+0xad
github.com/apache/yunikorn-k8shim/pkg/shim.TestApplicationScheduling()
/home/chenyulin0719/yunikorn/yunikorn-k8shim/pkg/shim/scheduler_test.go:60 +0x8c
testing.tRunner()
/usr/local/go/src/testing/testing.go:1689 +0x21e
testing.(*T).Run.gowrap1()
/usr/local/go/src/testing/testing.go:1742 +0x44
================== {code}
{*}Root cause{*}:
EventStore.events were read/write by 2 different goroutine.
* Goroutine 1: EventPublisher.StartService() call sp.store.CollectEvents(),
return reference of es.events[:es.idx]
* Goroutine 2: EventSystemImpl.StartServiceWithPublisher() call
ec.Store.Store(event) to update es.events[es.idx] = event
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org
