[ https://issues.apache.org/jira/browse/IGNITE-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Lang updated IGNITE-6506: ------------------------------ Description: I have a cluster with two nodes: A and B. On startup, node A and B wait for each other to be connected and then node A will attempt to activate the cluster. While testing high availability we find that if a node is stopped during the persistent store checkpoint, we cannot activate the cluster on startup without deleting the persistent storage directory. Specifically in the case where node A is stopped during checkpointing, upon the next startup it will encounter several exceptions during activation and then hang without completing activation. Here is the log. {noformat} 2017-09-26 12:11:24 [tcp-disco-msg-worker-#2%mbe%] INFO o.a.i.i.p.c.GridClusterStateProcessor - Start state transition: true 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.ignite.internal.exchange.time - Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], crd=true, evt=18, node=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], evtNode=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], customEvt=ChangeGlobalStateMessage [id=1d0cb2fbe51-7967bd11-40aa-40fe-b0a6-c43302cd4ee7, reqId=f7155dea-fede-4340-b244-7a3b65f167a8, initiatingNodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, activate=true]] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Start activation process [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.f.FilePageStoreManager - Resolved page store work directory: /opt/mbe1/ignite/db/mbe_MBE1 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log work directory: /opt/mbe1/ignite/db/wal/mbe_MBE1 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log archive directory: /opt/mbe1/ignite/db/wal/archive/mbe_MBE1 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - No user-defined default MemoryPolicy found; system default of 1GB size will be used. 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory [memoryAllocated=100.0 MiB, pages=48592, tableSize=2.9 MiB, checkpointBuffer=819.4 MiB] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory [memoryAllocated=3.1 GiB, pages=1544064, tableSize=91.0 MiB, checkpointBuffer=819.4 MiB] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin, end marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Checking memory state [lastValidPos=FileWALPointer [idx=139, fileOffset=31406805, len=20731, forceFlush=false], lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Ignite node stopped in the middle of checkpoint. Will restore memory state and finish checkpoint on node start. 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to activate node components [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]] java.lang.ArrayIndexOutOfBoundsException: -1 at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:82) at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:92) at org.apache.ignite.internal.pagemem.wal.record.delta.DataPageInsertRecord.applyDelta(DataPageInsertRecord.java:57) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1415) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:539) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:770) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:574) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], waitTime=0ms] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin, end marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT] org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Snapshot initialization completed [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], time=0ms] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.GridCachePartitionExchangeManager - Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], topVer=2, nodeId8=62cf0ccb, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1506445884063]], crd=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=1495762349], init=false, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], exchActions=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1506445884073, centralizedAff=false, changeGlobalStateE=java.lang.ArrayIndexOutOfBoundsException: -1, forcedRebFut=null, done=true, evtLatch=0, remaining=[bd81ae2b-91c0-45f1-b589-e73b7ffb56ee], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException: WAL history is too short [descs=[o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]], hash=927462353]] org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) {noformat} The thread activating the node is blocked at this point and the node cannot continue startup. was: I have a cluster with two nodes: A and B. On startup, node A and B wait for each other to be connected and then node A will attempt to activate the cluster. While testing high availability we find that if a node is stopped during the persistent store checkpoint, we cannot activate the cluster on startup without deleting the persistent storage directory. Specifically in the case where node A is stopped during checkpointing, upon the next startup it will encounter several exceptions during activation and then hang without completing activation. Here is the log. {noformat} 2017-09-26 12:11:24 [tcp-disco-msg-worker-#2%mbe%] INFO o.a.i.i.p.c.GridClusterStateProcessor - Start state transition: true 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.ignite.internal.exchange.time - Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], crd=true, evt=18, node=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], evtNode=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], customEvt=ChangeGlobalStateMessage [id=1d0cb2fbe51-7967bd11-40aa-40fe-b0a6-c43302cd4ee7, reqId=f7155dea-fede-4340-b244-7a3b65f167a8, initiatingNodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, activate=true]] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Start activation process [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.f.FilePageStoreManager - Resolved page store work directory: /opt/mbe1/ignite/db/mbe_MBE1 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log work directory: /opt/mbe1/ignite/db/wal/mbe_MBE1 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log archive directory: /opt/mbe1/ignite/db/wal/archive/mbe_MBE1 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - No user-defined default MemoryPolicy found; system default of 1GB size will be used. 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory [memoryAllocated=100.0 MiB, pages=48592, tableSize=2.9 MiB, checkpointBuffer=819.4 MiB] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory [memoryAllocated=3.1 GiB, pages=1544064, tableSize=91.0 MiB, checkpointBuffer=819.4 MiB] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin, end marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Checking memory state [lastValidPos=FileWALPointer [idx=139, fileOffset=31406805, len=20731, forceFlush=false], lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Ignite node stopped in the middle of checkpoint. Will restore memory state and finish checkpoint on node start. 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to activate node components [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]] java.lang.ArrayIndexOutOfBoundsException: -1 at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:82) at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:92) at org.apache.ignite.internal.pagemem.wal.record.delta.DataPageInsertRecord.applyDelta(DataPageInsertRecord.java:57) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1415) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:539) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:770) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:574) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], waitTime=0ms] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin, end marker = /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT] org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Snapshot initialization completed [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], time=0ms] 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR o.a.i.i.p.c.GridCachePartitionExchangeManager - Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], topVer=2, nodeId8=62cf0ccb, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1506445884063]], crd=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=1495762349], init=false, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], exchActions=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1506445884073, centralizedAff=false, changeGlobalStateE=java.lang.ArrayIndexOutOfBoundsException: -1, forcedRebFut=null, done=true, evtLatch=0, remaining=[bd81ae2b-91c0-45f1-b589-e73b7ffb56ee], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException: WAL history is too short [descs=[o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]], hash=927462353]] org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290) at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) {noformat} > Cluster activation hangs if a node was stopped during persistent storage > checkpoint > ----------------------------------------------------------------------------------- > > Key: IGNITE-6506 > URL: https://issues.apache.org/jira/browse/IGNITE-6506 > Project: Ignite > Issue Type: Bug > Components: persistence > Affects Versions: 2.2 > Reporter: Joel Lang > Priority: Critical > > I have a cluster with two nodes: A and B. > On startup, node A and B wait for each other to be connected and then node A > will attempt to activate the cluster. > While testing high availability we find that if a node is stopped during the > persistent store checkpoint, we cannot activate the cluster on startup > without deleting the persistent storage directory. Specifically in the case > where node A is stopped during checkpointing, upon the next startup it will > encounter several exceptions during activation and then hang without > completing activation. > Here is the log. > {noformat} > 2017-09-26 12:11:24 [tcp-disco-msg-worker-#2%mbe%] INFO > o.a.i.i.p.c.GridClusterStateProcessor - Start state transition: true > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.ignite.internal.exchange.time - Started exchange init > [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], crd=true, evt=18, > node=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, > addrs=[10.5.17.19, 127.0.0.1], > sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], > discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, > loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], > evtNode=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, > addrs=[10.5.17.19, 127.0.0.1], > sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], > discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063, > loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], > customEvt=ChangeGlobalStateMessage > [id=1d0cb2fbe51-7967bd11-40aa-40fe-b0a6-c43302cd4ee7, > reqId=f7155dea-fede-4340-b244-7a3b65f167a8, > initiatingNodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, activate=true]] > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Start activation process > [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, > topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]] > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.f.FilePageStoreManager - Resolved page store work directory: > /opt/mbe1/ignite/db/mbe_MBE1 > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log work > directory: /opt/mbe1/ignite/db/wal/mbe_MBE1 > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log archive > directory: /opt/mbe1/ignite/db/wal/archive/mbe_MBE1 > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN > o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - No user-defined default > MemoryPolicy found; system default of 1GB size will be used. > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory > [memoryAllocated=100.0 MiB, pages=48592, tableSize=2.9 MiB, > checkpointBuffer=819.4 MiB] > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory > [memoryAllocated=3.1 GiB, pages=1544064, tableSize=91.0 MiB, > checkpointBuffer=819.4 MiB] > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start > marker = > /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin, > end marker = > /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Checking memory state > [lastValidPos=FileWALPointer [idx=139, fileOffset=31406805, len=20731, > forceFlush=false], lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0, > forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505] > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN > o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Ignite node stopped in the > middle of checkpoint. Will restore memory state and finish checkpoint on node > start. > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR > o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to activate node > components [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false, > topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]] > java.lang.ArrayIndexOutOfBoundsException: -1 > at > org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:82) > at > org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:92) > at > org.apache.ignite.internal.pagemem.wal.record.delta.DataPageInsertRecord.applyDelta(DataPageInsertRecord.java:57) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1415) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:539) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:770) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:574) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:745) > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Finished waiting for > partition release future [topVer=AffinityTopologyVersion [topVer=2, > minorTopVer=1], waitTime=0ms] > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start > marker = > /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin, > end marker = > /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Applying lost cache updates > since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOffset=0, > len=0, forceFlush=false], > lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505] > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR > o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to reinitialize > local partitions (preloading will be stopped): GridDhtPartitionExchangeId > [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb, > evt=DISCOVERY_CUSTOM_EVT] > org.apache.ignite.IgniteCheckedException: WAL history is too short > [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], > start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]] > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375) > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350) > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290) > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:745) > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO > o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Snapshot initialization > completed [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], time=0ms] > 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR > o.a.i.i.p.c.GridCachePartitionExchangeManager - Failed to wait for completion > of partition map exchange (preloading will not start): > GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, > reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null, > affTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], > super=DiscoveryEvent [evtNode=TcpDiscoveryNode > [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], > sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], > discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, > loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], topVer=2, > nodeId8=62cf0ccb, msg=null, type=DISCOVERY_CUSTOM_EVT, > tstamp=1506445884063]], crd=TcpDiscoveryNode > [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1], > sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510], > discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308, > loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], > exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, > minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT], added=true, > initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, > hash=1495762349], init=false, lastVer=null, partReleaseFut=GridCompoundFuture > [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, > futs=[true, true, true, true]], exchActions=null, affChangeMsg=null, > skipPreload=false, clientOnlyExchange=false, initTs=1506445884073, > centralizedAff=false, > changeGlobalStateE=java.lang.ArrayIndexOutOfBoundsException: -1, > forcedRebFut=null, done=true, evtLatch=0, > remaining=[bd81ae2b-91c0-45f1-b589-e73b7ffb56ee], super=GridFutureAdapter > [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException: > WAL history is too short > [descs=[o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, > > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], > start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]], > hash=927462353]] > org.apache.ignite.IgniteCheckedException: WAL history is too short > [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a, > > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b], > start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]] > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375) > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350) > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290) > at > org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:745) > {noformat} > The thread activating the node is blocked at this point and the node cannot > continue startup. -- This message was sent by Atlassian JIRA (v6.4.14#64029)