[ https://issues.apache.org/jira/browse/IGNITE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495152#comment-16495152 ]
Alexey Kuznetsov commented on IGNITE-6538: ------------------------------------------ The exception that led test to hang : {code:java} [org.apache.ignite:ignite-core] java.lang.AssertionError: calculatedOffset=3072, allocated=2048, headerSize=1024 [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:346) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:341) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:325) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:142) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:167) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.<init>(AbstractFreeList.java:370) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage$FreeListImpl.<init>(MetaStorage.java:472) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.init(MetaStorage.java:139) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:808) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:901) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [13:46:39]W: [org.apache.ignite:ignite-core] at java.lang.Thread.run(Thread.java:748) [13:46:39] : [Step 3/4] [2018-05-22 10:46:39,411][INFO ][exchange-worker-#201982%node0-backUp%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=6, minorTopVer=1], resVer=null, err=java.lang.AssertionError: calculatedOffset=3072, allocated=2048, headerSize=1024] [13:46:39]W: [org.apache.ignite:ignite-core] [2018-05-22 10:46:39,415][ERROR][exchange-worker-#201982%node0-backUp%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.NoOpFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.AssertionError: calculatedOffset=3072, allocated=2048, headerSize=1024]] [13:46:39]W: [org.apache.ignite:ignite-core] java.lang.AssertionError: calculatedOffset=3072, allocated=2048, headerSize=1024 [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:346) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:341) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:325) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:142) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:167) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.<init>(AbstractFreeList.java:370) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage$FreeListImpl.<init>(MetaStorage.java:472) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.init(MetaStorage.java:139) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:808) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:901) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332) [13:46:39]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [13:46:39]W: [org.apache.ignite:ignite-core] at java.lang.Thread.run(Thread.java:748) ... Failed to acquire file lock (local nodeId:7ee5a633-993e-4430-8e93-2bfd3484fe9f, already locked by [effc88b6-b8b5-43a2-8db5-be2e00d6a60a][127.0.0.1, 172.17.0.7][TCP:10802, TCP:45012, TCP:47502]), will try again in 1s: /data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/node2/lock ... Failed to acquire file lock during 10 sec, (locked by [effc88b6-b8b5-43a2-8db5-be2e00d6a60a][127.0.0.1, 172.17.0.7][TCP:10802, TCP:45012, TCP:47502]): /data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/node2/lock [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.tryLock(GridCacheDatabaseSharedManager.java:4700) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.lock(GridCacheDatabaseSharedManager.java:922) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:250) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:888) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [13:46:49]W: [org.apache.ignite:ignite-core] at java.lang.Thread.run(Thread.java:748) [13:46:49]W: [org.apache.ignite:ignite-core] [2018-05-22 10:46:49,297][ERROR][exchange-worker-#202059%node1-backUp%][GridDhtPartitionsExchangeFuture] Failed to activate node components [nodeId=f6f3d53b-7ee3-42c5-add7-9c58066f5f35, client=false, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=1]] [13:46:49]W: [org.apache.ignite:ignite-core] class org.apache.ignite.IgniteCheckedException: Failed to acquire file lock during 10 sec, (locked by [c859c9b3-b57d-44db-bdb8-778302f3d16c][127.0.0.1, 172.17.0.7][TCP:10801, TCP:45011, TCP:47501]): /data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/node1/lock [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.tryLock(GridCacheDatabaseSharedManager.java:4700) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.lock(GridCacheDatabaseSharedManager.java:922) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:250) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:888) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648) [13:46:49] : [Step 3/4] [2018-05-22 10:46:49,299][INFO ][exchange-worker-#202136%node2-backUp%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=1], waitTime=0ms, futInfo=NA] [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332) [13:46:49]W: [org.apache.ignite:ignite-core] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [13:46:49]W: [org.apache.ignite:ignite-core] at java.lang.Thread.run(Thread.java:748) {code} > Ignite Activate/Deactivate Cluster suite: After tests validation improvements > test became flaky on TC > ----------------------------------------------------------------------------------------------------- > > Key: IGNITE-6538 > URL: https://issues.apache.org/jira/browse/IGNITE-6538 > Project: Ignite > Issue Type: Bug > Components: persistence > Affects Versions: 2.1 > Reporter: Dmitriy Pavlov > Assignee: Alexey Kuznetsov > Priority: Major > Labels: MakeTeamcityGreenAgain, Muted_test > Fix For: 2.6 > > > https://ci.ignite.apache.org/project.html?projectId=Ignite20Tests&testNameId=-238119157387028099&tab=testDetails&branch_Ignite20Tests=%3Cdefault%3E > {code} > [14:36:09]W: [org.apache.ignite:ignite-core] [2018-05-22 > 11:36:09,817][WARN > ][exchange-worker-#202059%node1-backUp%][GridDhtPartitionsExchangeFuture] > Unable to await partitions release latch within timeout: ClientLatch > [coordinator=TcpDiscoveryNode [id=ba7f90d8-3a1f-488a-a8ea-3ae2fdf016fe, > addrs=[127.0.0.1, 172.17.0.7], sockAddrs=[/127.0.0.1:47503, > d377975c5791/172.17.0.7:47503], discPort=47503, order=1, intOrder=1, > lastExchangeTime=1526985998704, loc=false, ver=2.5.0#20180522-sha1:d37470c7, > isClient=false], ackSent=true, super=CompletableLatch [id=exchange, > topVer=AffinityTopologyVersion [topVer=6, minorTopVer=1]]] > {code} > Fail test: > {code} > org.apache.ignite.internal.processors.cache.persistence.standbycluster.IgniteChangeGlobalStateTest#testActivateAfterFailGetLock > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)