[
https://issues.apache.org/jira/browse/IGNITE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexei Scherbakov resolved IGNITE-6491.
---------------------------------------
Resolution: Fixed
Fix Version/s: (was: 2.2)
2.3
> Race in TopologyValidator.validate() and EVT_NODE_LEFT listener calls
> (split-brain activator)
> ---------------------------------------------------------------------------------------------
>
> Key: IGNITE-6491
> URL: https://issues.apache.org/jira/browse/IGNITE-6491
> Project: Ignite
> Issue Type: Bug
> Components: cache, general
> Affects Versions: 2.1
> Reporter: Alexandr Kuramshin
> Assignee: Alexandr Kuramshin
> Fix For: 2.3
>
>
> The following wrong cache {{validate}}/{{put}} sequence may occur
> On node left {{GridDhtPartitionsExchangeFuture}} will be generated by the
> {{disco-event-worker}} thread.
> Then the {{exchange-worker}} thread does
> {noformat}
> Split-brain detected [cacheName=test40, activatorTopVer=0, cacheTopVer=14]
> at
> org.apache.ignite.internal.util.IgniteUtils.dumpStack(IgniteUtils.java:1141)
> at
> org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator.validate(IgniteTopologyValidatorGridSplitCacheTest.java:307)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTopologyFutureAdapter.validateCacheGroup(GridDhtTopologyFutureAdapter.java:64)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1456)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:115)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:450)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:668)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2278)
> {noformat}
> The result of validation is stored in {{grpValidRes}} with value of {{false}}.
> After some delay the {{disco-event-worker}} thread will do
> {noformat}
> java.lang.Exception: Node is segment activator [cacheName=test40,
> activatorTopVer=14]
> at
> org.apache.ignite.internal.util.IgniteUtils.dumpStack(IgniteUtils.java:1141)
> at
> org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator$2.apply(IgniteTopologyValidatorGridSplitCacheTest.java:360)
> at
> org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest$SplitAwareTopologyValidator$2.apply(IgniteTopologyValidatorGridSplitCacheTest.java:349)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1463)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:859)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:844)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:341)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:307)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:2478)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2684)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2507)
> {noformat}
> After this invocation the result of {{SplitAwareTopologyValidator.validate}}
> should be changed to {{true}}, but it was already invoked and the result has
> been cached in {{grpValidRes}} with the value of {{false}}.
> So any successive calls to {{cache.put}} causes to fail
> {noformat}
> Test failed.
> java.lang.RuntimeException: tryPut() failed
> [gridName=cache.IgniteTopologyValidatorGridSplitCacheTest0]
> at
> org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest.tryPut(IgniteTopologyValidatorGridSplitCacheTest.java:262)
> at
> org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest.testTopologyValidator(IgniteTopologyValidatorGridSplitCacheTest.java:182)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.cache.CacheException: class
> org.apache.ignite.IgniteCheckedException: Failed to perform cache operation
> (cache topology is not valid): test40
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1327)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1672)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1032)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:872)
> at
> org.apache.ignite.internal.processors.cache.IgniteTopologyValidatorGridSplitCacheTest.tryPut(IgniteTopologyValidatorGridSplitCacheTest.java:252)
> ... 10 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to perform
> cache operation (cache topology is not valid): test40
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTopologyFutureAdapter.validateCache(GridDhtTopologyFutureAdapter.java:112)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicSingleUpdateFuture.mapOnTopology(GridNearAtomicSingleUpdateFuture.java:415)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update0(GridDhtAtomicCache.java:1170)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:659)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2334)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2311)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1029)
> ... 12 more
> {noformat}
> The updated test {{IgniteTopologyValidatorGridSplitCacheTest}} fails
> frequently on my laptop with 8 nodes and 100 caches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
