[
https://issues.apache.org/jira/browse/IGNITE-8768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507330#comment-16507330
]
Eduard Shangareev commented on IGNITE-8768:
-------------------------------------------
Test which caused JVM Crash -
IgniteClusterActivateDeactivateTestWithPersistence#testDeactivateDuringEvictionAndRebalance.
Last interesting logs:
{code}
[2018-06-08 22:39:56,398][INFO
][sys-#23017%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridCacheProcessor]
Stopped cache [cacheName=default]
[2018-06-08 22:39:56,401][INFO
][sys-#23017%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridCacheProcessor]
Stopped cache [cacheName=ignite-sys-cache]
[2018-06-08
22:39:56,401][ERROR][sys-#23004%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridDhtPartitionsEvictor]
Partition eviction failed, this can cause grid hang.
class org.apache.ignite.IgniteException: Failed to find cache (cache was not
started yet or cache was already stopped): default
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.partition(GridCacheAffinityManager.java:176)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.partition(GridCacheAffinityManager.java:160)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.<init>(GridDhtCacheEntry.java:90)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1.create(GridDhtLocalPartition.java:85)
at
org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMapImpl.java:113)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1011)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:859)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call(GridDhtPartitionsEvictor.java:95)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call(GridDhtPartitionsEvictor.java:73)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6710)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
][INFO
][sys-#23017%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridDhtPartitionDemander]
Cancelled rebalancing from all nodes [topology=AffinityTopologyVersion
[topVer=5, minorTopVer=0]]
][INFO
][sys-#23017%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridDhtPartitionDemander]
Completed rebalance future: RebalanceFuture [grp=CacheGroupContext
[grp=default], topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0],
rebalanceId=1]
][INFO
][db-checkpoint-thread-#22994%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=aab71982-6199-49ad-80e2-e16d36ec14b0,
startPtr=FileWALPointer [idx=0, fileOff=30675421, len=41],
checkpointLockWait=0ms, checkpointLockHoldTime=0ms,
walCpRecordFsyncDuration=0ms, pages=120, reason='timeout']
[2018-06-08
22:39:56,458][ERROR][sys-#23004%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][GridDhtPartitionsEvictor]
Partition eviction failed, this can cause grid hang.
class org.apache.ignite.IgniteException: Runtime failure on bounds:
[lower=null, upper=null]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:1506)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.cursor(GridCacheOffheapManager.java:1630)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$6.onHasNext(IgniteCacheOffheapManagerImpl.java:662)
at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:996)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:859)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call(GridDhtPartitionsEvictor.java:95)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call(GridDhtPartitionsEvictor.java:73)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6710)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: Failed to get page store
for the given cache ID (cache has not been started): 1544803905
at
org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:57)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:73)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:146)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:41)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4660)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4562)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4501)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:927)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:959)
... 16 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to get page
store for the given cache ID (cache has not been started): 1544803905
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.getStore(FilePageStoreManager.java:898)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:361)
at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:348)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:140)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
at
org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:54)
[01:39:56]W: [org.apache.ignite:ignite-core] ... 24
more
{code}
Thread which caused crash:
{code}
"sys-#23004%cache.IgniteClusterActivateDeactivateTestWithPersistence2%"
{code}
Its stach trace from core-dump:
{code}
Stack: [0x00007ff2d26e7000,0x00007ff2d27e8000], sp=0x00007ff2d27e5c70, free
space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 23611 C2
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(IJZ)J
(1306 bytes) @ 0x00007ff66e599b58 [0x00007ff66e5996c0+0x498]
J 9610 C2
org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(J)J
(104 bytes) @ 0x00007ff66ef2ae48 [0x00007ff66ef2adc0+0x88]
j
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(Ljava/lang/Object;Ljava/lang/Object;)Lorg/apache/ignite/internal/util/lang/GridCursor;+17
J 23614 C1
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Lorg/apache/ignite/internal/util/lang/GridCursor;
(196 bytes) @ 0x00007ff6705f048c [0x00007ff6705f0380+0x10c]
J 23023 C2
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(Ljava/lang/Object;Ljava/lang/Object;)Lorg/apache/ignite/internal/util/lang/GridCursor;
(8 bytes) @ 0x00007ff66e36af54 [0x00007ff66e36af20+0x34]
j
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor()Lorg/apache/ignite/internal/util/lang/GridCursor;+6
J 24555 C2
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$6.onHasNext()Z
(151 bytes) @ 0x00007ff6714fb310 [0x00007ff6714fa6a0+0xc70]
J 7360 C2
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX()Z (20
bytes) @ 0x00007ff66e75964c [0x00007ff66e7595e0+0x6c]
j org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext()Z+1
j
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.clearAll()J+151
j
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition.tryClear()Z+44
j
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call()Ljava/lang/Boolean;+186
j
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtPartitionsEvictor$1.call()Ljava/lang/Object;+1
J 20750 C1
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(Ljava/lang/ClassLoader;Ljava/util/concurrent/Callable;)Ljava/lang/Object;
(57 bytes) @ 0x00007ff66f96731c [0x00007ff66f966680+0xc9c]
J 21885 C1
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body()V
(92 bytes) @ 0x00007ff670824b4c [0x00007ff670824a80+0xcc]
J 20287 C2 org.apache.ignite.internal.util.worker.GridWorker.run()V (1232
bytes) @ 0x00007ff6707de6f0 [0x00007ff6707de500+0x1f0]
J 22515 C2
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(225 bytes) @ 0x00007ff670e03728 [0x00007ff670e034c0+0x268]
J 12836 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @
0x00007ff66f65bbc4 [0x00007ff66f65bac0+0x104]
J 17745 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007ff66f7b7b28
[0x00007ff66f7b7ae0+0x48]
{code}
> JVM crash in PDS1 suite in master branch
> ----------------------------------------
>
> Key: IGNITE-8768
> URL: https://issues.apache.org/jira/browse/IGNITE-8768
> Project: Ignite
> Issue Type: Bug
> Components: persistence
> Reporter: Sergey Chugunov
> Priority: Blocker
> Labels: MakeTeamcityGreenAgain
> Fix For: 2.6
>
>
> JVM crash in latest build: [TC
> link|https://ci.ignite.apache.org/viewLog.html?buildId=1372456&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_Pds1]
> It is the first crash is latest 15 builds: [TC
> link|https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Pds1&branch_IgniteTests24Java8=%3Cdefault%3E&tab=buildTypeStatusDiv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
