Hello! It seems that the node fell behind in switching to new topology. Can you provide full log of this node? I guess that something was blocking PME.
Regards, -- Ilya Kasnacheev чт, 9 июл. 2020 г. в 09:44, Mahesh Renduchintala < mahesh.renduchint...@aline-consulting.com>: > Hi, > > We have a crash in our environment with the below error. > Any insight into what might have gone wrong? > > regards > Mahesh > > > ^-- Heap [used=17211MB, free=64.98%, comm=49152MB] > ^-- Off-heap [used=48448MB, free=26.41%, comm=65736MB] > ^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB] > ^-- default region [used=48447MB, free=26.07%, comm=65536MB] > ^-- metastoreMemPlc region [used=0MB, free=99.21%, comm=0MB] > ^-- TxLog region [used=0MB, free=100%, comm=100MB] > ^-- Ignite persistence [used=48647MB] > ^-- sysMemPlc region [used=0MB] > ^-- default region [used=48646MB] > ^-- metastoreMemPlc region [used=0MB] > ^-- TxLog region [used=0MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=4, qSize=0] > ^-- System thread pool [active=11, idle=12, qSize=0] > [18:44:06,313][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] > Finish exchange future [startVer=AffinityTopologyVersion [topVer=83, > minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], > err=null, rebalanced=true, wasRebalanced=true] > [18:44:06,316][WARNING][sys-stripe-21-#22][finish] Received finish request > for completed transaction (the message may be too late) > [txId=GridCacheVersion [topVer=205532475, order=1594223109470, > nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, > commit=false] > [18:44:06,317][WARNING][sys-stripe-18-#19][finish] Received finish request > for completed transaction (the message may be too late) > [txId=GridCacheVersion [topVer=205532475, order=1594223109273, > nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, > commit=false] > [18:44:06,317][WARNING][sys-stripe-23-#24][finish] Received finish request > for completed transaction (the message may be too late) > [txId=GridCacheVersion [topVer=205532475, order=1594223109436, > nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, > commit=false] > [18:44:06,321][WARNING][sys-stripe-21-#22][finish] Received finish request > for completed transaction (the message may be too late) > [txId=GridCacheVersion [topVer=205532475, order=1594223109502, > nodeOrder=51], dhtTxId=null, node=35c37b55-ec30-4457-b861-403bcfc20c12, > commit=false] > [18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] > Completed partition exchange > [localNode=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=83, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode > [id=c19d4735-2b52-487f-9d6d-0574b8f15858, > consistentId=L1APISERVICE_7aa60a92371, addrs=ArrayList [10.244.0.93, > 127.0.0.1], sockAddrs=HashSet [/10.244.0.93:0, /127.0.0.1:0], discPort=0, > order=68, intOrder=39, lastExchangeTime=1594131247817, loc=false, > ver=2.8.1#20200521-sha1:86422096, isClient=true], done=true, > newCrdFut=null], topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0]] > [18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] > Exchange timings [startVer=AffinityTopologyVersion [topVer=83, > minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], > stage="Waiting in exchange queue" (0 ms), stage="Exchange parameters > initialization" (0 ms), stage="Determine exchange type" (16 ms), > stage="Exchange done" (71053 ms), stage="Total time" (71069 ms)] > [18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture] > Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=83, > minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0]] > [18:44:06,327][INFO][exchange-worker-#70][time] Finished exchange init > [topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], crd=false] > [18:44:06,345][INFO][db-checkpoint-thread-#112][GridCacheDatabaseSharedManager] > Checkpoint started [checkpointId=92718d25-2db4-4691-886c-ec26b8b6ecba, > startPtr=FileWALPointer [idx=389, fileOff=2151853, len=17162371], > checkpointBeforeLockTime=869ms, checkpointLockWait=86163ms, > checkpointListenersExecuteTime=784ms, checkpointLockHoldTime=859ms, > walCpRecordFsyncDuration=20ms, writeCheckpointEntryDuration=1ms, > splitAndSortCpPagesDuration=24ms, pages=28028, reason='timeout'] > [18:44:06,388][SEVERE][sys-stripe-3-#4][] Critical system error detected. > Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], > failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=c*lass > o.a.i.IgniteException: Failed to resolve nodes topology > [*cacheGrp=SQL_PUBLIC_MILLION, > topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0], > history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], > AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion > [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, > minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], > AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion > [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, > minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], > AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot > [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], > locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, > consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList > [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, > order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, > ver=2.8.1#20200521-sha1:86422096, isClient=false]]]] > class org.apache.ignite.IgniteException: Failed to resolve nodes topology > [cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82, > minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, > minorTopVer=0], AffinityTopologyVersion [topVer=85, minorTopVer=0], > AffinityTopologyVersion [topVer=86, minorTopVer=0], AffinityTopologyVersion > [topVer=87, minorTopVer=0], AffinityTopologyVersion [topVer=88, > minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0], > AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion > [topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, > minorTopVer=0], AffinityTopologyVersion [topVer=93, minorTopVer=0]], > snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], > locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, > consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList > [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, > order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, > ver=2.8.1#20200521-sha1:86422096, isClient=false]] > at > org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999) > at > org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.lambda$null$0(IgniteTxHandler.java:645) > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:745) > [18:44:06,404][SEVERE][sys-stripe-14-#15][GridCacheIoManager] Failed > processing message [senderId=35c37b55-ec30-4457-b861-403bcfc20c12, > msg=GridNearTxPrepareRequest > [futId=62e33fa2371-a9969979-f2fb-410a-8d90-73efb0e13b0c, miniId=1, > topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0], > subjId=35c37b55-ec30-4457-b861-403bcfc20c12, taskNameHash=0, txLbl=null, > flags=, super=GridDistributedTxPrepareRequest [threadId=4722, > concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, > writeVer=GridCacheVersion [topVer=205532486, order=1594223109704, > nodeOrder=51], timeout=0, reads=ArrayList [], writes=ArrayList > [IgniteTxEntry [txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=72, > val=GridCacheInternalKeyImpl [name=lt_642138, > grpName=default-volatile-ds-group], hasValBytes=true], cacheId=1968300681], > val=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=], > prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], > entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, > explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], > filtersPassed=false, filtersSet=false, entry=null, prepared=0, > locked=false, nodeId=null, locMapped=false, expiryPlc=null, > transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, > xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=IgniteTxStateImpl > [activeCacheIds=[1968300681], recovery=false, mvccEnabled=false, > mvccCachingCacheIds=[], txMap=ArrayList [IgniteTxEntry [txKey=IgniteTxKey > [key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl > [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], > cacheId=1968300681], val=CacheObjectImpl [val=null, > hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null], > oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, > conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, > filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, > entry=GridDhtCacheEntry [rdrs=ReaderId[] [], part=72, > super=GridDistributedCacheEntry [super=GridCacheMapEntry > [key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl > [name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true], > val=CacheObjectImpl [val=null, hasValBytes=true], ver=GridCacheVersion > [topVer=205532475, order=1594223109257, nodeOrder=4], hash=-1064371155, > extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc [locs=LinkedList > [GridCacheMvccCandidate [nodeId=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, > ver=GridCacheVersion [topVer=205532486, order=1594223109711, nodeOrder=4], > threadId=4722, id=56494472, topVer=AffinityTopologyVersion [topVer=83, > minorTopVer=0], reentry=null, > otherNodeId=35c37b55-ec30-4457-b861-403bcfc20c12, otherVer=GridCacheVersion > [topVer=205532486, order=1594223109704, nodeOrder=51], mappedDhtNodes=null, > mappedNearNodes=null, ownerVer=null, serOrder=null, key=KeyCacheObjectImpl > [part=72, val=GridCacheInternalKeyImpl [name=lt_642138, > grpName=default-volatile-ds-group], hasValBytes=true], > masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0, > prevVer=null, nextVer=null]], rmts=null]], flags=2]]], prepared=0, > locked=false, nodeId=null, locMapped=false, expiryPlc=null, > transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null, > xidVer=GridCacheVersion [topVer=205532486, order=1594223109711, > nodeOrder=4]]]], flags=onePhase|last|sys, super=GridDistributedBaseMessage > [ver=GridCacheVersion [topVer=205532486, order=1594223109704, > nodeOrder=51], committedVers=null, rolledbackVers=null, cnt=0, > super=GridCacheIdMessage [cacheId=0, super=GridCacheMessage > [msgId=11018564, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion > [topVer=81, minorTopVer=44], err=null, skipPrepare=false]]]]]] > class org.apache.ignite.IgniteException: Failed to resolve nodes topology > [cacheGrp=N/A, topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], > history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], > AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion > [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, > minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], > AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion > [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, > minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], > AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot > [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], > locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, > consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList > [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, > order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, > ver=2.8.1#20200521-sha1:86422096, isClient=false]] > at > org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999) > at > org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.node(GridDiscoveryManager.java:1821) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1056) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:411) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:577) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:183) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367) > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:745) > [18:44:06,665][WARNING][sys-stripe-3-#4][CacheDiagnosticManager] Page > locks dump: > > Thread=[name=auth-#69, id=119], state=WAITING > Locked pages = [] > Locked pages log: name=auth-#69 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=checkpoint-runner-#113, id=167], state=RUNNABLE > Locked pages = [] > Locked pages log: name=checkpoint-runner-#113 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=checkpoint-runner-#114, id=168], state=RUNNABLE > Locked pages = [] > Locked pages log: name=checkpoint-runner-#114 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=checkpoint-runner-#115, id=169], state=RUNNABLE > Locked pages = [] > Locked pages log: name=checkpoint-runner-#115 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=checkpoint-runner-#116, id=170], state=RUNNABLE > Locked pages = [] > Locked pages log: name=checkpoint-runner-#116 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=client-connector-#6594, id=6902], state=WAITING > Locked pages = [] > Locked pages log: name=client-connector-#6594 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=client-connector-#6595, id=6903], state=WAITING > Locked pages = [] > Locked pages log: name=client-connector-#6595 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=client-connector-#6598, id=6906], state=WAITING > Locked pages = [] > Locked pages log: name=client-connector-#6598 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=client-connector-#6599, id=6907], state=WAITING > Locked pages = [] > Locked pages log: name=client-connector-#6599 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=client-connector-#6602, id=6911], state=WAITING > Locked pages = [] > Locked pages log: name=client-connector-#6602 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=client-connector-#6603, id=6912], state=WAITING > Locked pages = [] > Locked pages log: name=client-connector-#6603 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=client-connector-#6606, id=6915], state=WAITING > Locked pages = [] > Locked pages log: name=client-connector-#6606 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=client-connector-#6607, id=6916], state=WAITING > Locked pages = [] > Locked pages log: name=client-connector-#6607 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=db-checkpoint-thread-#112, id=166], state=WAITING > Locked pages = [] > Locked pages log: name=db-checkpoint-thread-#112 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=dms-writer-thread-#76, id=128], state=WAITING > Locked pages = [] > Locked pages log: name=dms-writer-thread-#76 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > Thread=[name=exchange-worker-#70, id=122], state=RUNNABLE > Locked pages = [] > Locked pages log: name=exchange-worker-#70 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=main, id=1], state=WAITING > Locked pages = [] > Locked pages log: name=main time=(1594233846390, 2020-07-08 18:44:06.390) > > > Thread=[name=sys-stripe-0-#1, id=20], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-0-#1 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-1-#2, id=21], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-1-#2 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-10-#11, id=30], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-10-#11 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-11-#12, id=31], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-11-#12 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-12-#13, id=32], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-12-#13 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-13-#14, id=33], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-13-#14 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-14-#15, id=34], state=RUNNABLE > Locked pages = [] > Locked pages log: name=sys-stripe-14-#15 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-15-#16, id=35], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-15-#16 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-16-#17, id=36], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-16-#17 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-17-#18, id=37], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-17-#18 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-18-#19, id=38], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-18-#19 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-19-#20, id=39], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-19-#20 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-2-#3, id=22], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-2-#3 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-20-#21, id=40], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-20-#21 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-21-#22, id=41], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-21-#22 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-22-#23, id=42], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-22-#23 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-23-#24, id=43], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-23-#24 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-24-#25, id=44], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-24-#25 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-25-#26, id=45], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-25-#26 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-26-#27, id=46], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-26-#27 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-27-#28, id=47], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-27-#28 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-28-#29, id=48], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-28-#29 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-29-#30, id=49], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-29-#30 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-3-#4, id=23], state=RUNNABLE > Locked pages = [] > Locked pages log: name=sys-stripe-3-#4 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-30-#31, id=50], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-30-#31 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-31-#32, id=51], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-31-#32 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-4-#5, id=24], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-4-#5 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-5-#6, id=25], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-5-#6 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-6-#7, id=26], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-6-#7 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-7-#8, id=27], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-7-#8 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-8-#9, id=28], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-8-#9 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=sys-stripe-9-#10, id=29], state=WAITING > Locked pages = [] > Locked pages log: name=sys-stripe-9-#10 time=(1594233846390, 2020-07-08 > 18:44:06.390) > > > Thread=[name=ttl-cleanup-worker-#108, id=160], state=TIMED_WAITING > Locked pages = [] > Locked pages log: name=ttl-cleanup-worker-#108 time=(1594233846390, > 2020-07-08 18:44:06.390) > > > > [18:44:06,666][SEVERE][sys-stripe-3-#4][] JVM will be halted immediately > due to the failure: [failureCtx=FailureContext > [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to > resolve nodes topology [cacheGrp=SQL_PUBLIC_MILLION, > topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0], > history=[AffinityTopologyVersion [topVer=84, minorTopVer=0], > AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion > [topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, > minorTopVer=0], AffinityTopologyVersion [topVer=88, minorTopVer=0], > AffinityTopologyVersion [topVer=89, minorTopVer=0], AffinityTopologyVersion > [topVer=90, minorTopVer=0], AffinityTopologyVersion [topVer=91, > minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0], > AffinityTopologyVersion [topVer=93, minorTopVer=0]], snap=Snapshot > [topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0]], > locNode=TcpDiscoveryNode [id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62, > consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList > [192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, > order=4, intOrder=3, lastExchangeTime=1594054666300, loc=true, > ver=2.8.1#20200521-sha1:86422096, isClient=false]]]] > >
