Hi,
We have a crash in our environment with the below error.
Any insight into what might have gone wrong?
regards
Mahesh
^-- Heap [used=17211MB, free=64.98%, comm=49152MB]
^-- Off-heap [used=48448MB, free=26.41%, comm=65736MB]
^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB]
^-- default region [used=48447MB, free=26.07%, comm=65536MB]
^-- metastoreMemPlc region [used=0MB, free=99.21%, comm=0MB]
^-- TxLog region [used=0MB, free=100%, comm=100MB]
^-- Ignite persistence [used=48647MB]
^-- sysMemPlc region [used=0MB]
^-- default region [used=48646MB]
^-- metastoreMemPlc region [used=0MB]
^-- TxLog region [used=0MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=4, qSize=0]
^-- System thread pool [active=11, idle=12, qSize=0]
[18:44:06,313][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=83,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0],
err=null, rebalanced=true, wasRebalanced=true]
[18:44:06,316][WARNING][sys-stripe-21-#22][finish] Received finish request for
completed transaction (the message may be too late) [txId=GridCacheVersion
[topVer=205532475, order=1594223109470, nodeOrder=51], dhtTxId=null,
node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,317][WARNING][sys-stripe-18-#19][finish] Received finish request for
completed transaction (the message may be too late) [txId=GridCacheVersion
[topVer=205532475, order=1594223109273, nodeOrder=51], dhtTxId=null,
node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,317][WARNING][sys-stripe-23-#24][finish] Received finish request for
completed transaction (the message may be too late) [txId=GridCacheVersion
[topVer=205532475, order=1594223109436, nodeOrder=51], dhtTxId=null,
node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,321][WARNING][sys-stripe-21-#22][finish] Received finish request for
completed transaction (the message may be too late) [txId=GridCacheVersion
[topVer=205532475, order=1594223109502, nodeOrder=51], dhtTxId=null,
node=35c37b55-ec30-4457-b861-403bcfc20c12, commit=false]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture]
Completed partition exchange [localNode=d38d1293-9dd5-4b4e-9934-97ba3fbafa62,
exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
[topVer=83, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode
[id=c19d4735-2b52-487f-9d6d-0574b8f15858,
consistentId=L1APISERVICE_7aa60a92371, addrs=ArrayList [10.244.0.93,
127.0.0.1], sockAddrs=HashSet [/10.244.0.93:0, /127.0.0.1:0], discPort=0,
order=68, intOrder=39, lastExchangeTime=1594131247817, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], done=true, newCrdFut=null],
topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0]]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture]
Exchange timings [startVer=AffinityTopologyVersion [topVer=83, minorTopVer=0],
resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], stage="Waiting in
exchange queue" (0 ms), stage="Exchange parameters initialization" (0 ms),
stage="Determine exchange type" (16 ms), stage="Exchange done" (71053 ms),
stage="Total time" (71069 ms)]
[18:44:06,327][INFO][exchange-worker-#70][GridDhtPartitionsExchangeFuture]
Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=83,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=83, minorTopVer=0]]
[18:44:06,327][INFO][exchange-worker-#70][time] Finished exchange init
[topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0], crd=false]
[18:44:06,345][INFO][db-checkpoint-thread-#112][GridCacheDatabaseSharedManager]
Checkpoint started [checkpointId=92718d25-2db4-4691-886c-ec26b8b6ecba,
startPtr=FileWALPointer [idx=389, fileOff=2151853, len=17162371],
checkpointBeforeLockTime=869ms, checkpointLockWait=86163ms,
checkpointListenersExecuteTime=784ms, checkpointLockHoldTime=859ms,
walCpRecordFsyncDuration=20ms, writeCheckpointEntryDuration=1ms,
splitAndSortCpPagesDuration=24ms, pages=28028, reason='timeout']
[18:44:06,388][SEVERE][sys-stripe-3-#4][] Critical system error detected. Will
be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler
[tryStop=false, timeout=0, super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to
resolve nodes topology [cacheGrp=SQL_PUBLIC_MILLION,
topVer=AffinityTopologyVersion [topVer=82, minorTopVer=0],
history=[AffinityTopologyVersion [topVer=84, minorTopVer=0],
AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion
[topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0],
AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion
[topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0],
AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion
[topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93,
minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93,
minorTopVer=0]], locNode=TcpDiscoveryNode
[id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62,
consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList
[192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4,
intOrder=3, lastExchangeTime=1594054666300, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology
[cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82,
minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0],
AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion
[topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0],
AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion
[topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0],
AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion
[topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93,
minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93,
minorTopVer=0]], locNode=TcpDiscoveryNode
[id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62,
consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList
[192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4,
intOrder=3, lastExchangeTime=1594054666300, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.cacheGroupAffinityNodes(GridDiscoveryManager.java:1881)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.needRemap(IgniteTxHandler.java:744)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:458)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:176)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.lambda$null$0(IgniteTxHandler.java:645)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[18:44:06,404][SEVERE][sys-stripe-14-#15][GridCacheIoManager] Failed processing
message [senderId=35c37b55-ec30-4457-b861-403bcfc20c12,
msg=GridNearTxPrepareRequest
[futId=62e33fa2371-a9969979-f2fb-410a-8d90-73efb0e13b0c, miniId=1,
topVer=AffinityTopologyVersion [topVer=93, minorTopVer=0],
subjId=35c37b55-ec30-4457-b861-403bcfc20c12, taskNameHash=0, txLbl=null,
flags=, super=GridDistributedTxPrepareRequest [threadId=4722,
concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, writeVer=GridCacheVersion
[topVer=205532486, order=1594223109704, nodeOrder=51], timeout=0,
reads=ArrayList [], writes=ArrayList [IgniteTxEntry [txKey=IgniteTxKey
[key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138,
grpName=default-volatile-ds-group], hasValBytes=true], cacheId=1968300681],
val=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=],
prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [],
filtersPassed=false, filtersSet=false, entry=null, prepared=0, locked=false,
nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0,
partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0,
plc=2, txState=IgniteTxStateImpl [activeCacheIds=[1968300681], recovery=false,
mvccEnabled=false, mvccCachingCacheIds=[], txMap=ArrayList [IgniteTxEntry
[txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=72,
val=GridCacheInternalKeyImpl [name=lt_642138,
grpName=default-volatile-ds-group], hasValBytes=true], cacheId=1968300681],
val=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=],
prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null,
explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [],
filtersPassed=false, filtersSet=false, entry=GridDhtCacheEntry [rdrs=ReaderId[]
[], part=72, super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl [name=lt_642138,
grpName=default-volatile-ds-group], hasValBytes=true], val=CacheObjectImpl
[val=null, hasValBytes=true], ver=GridCacheVersion [topVer=205532475,
order=1594223109257, nodeOrder=4], hash=-1064371155,
extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc [locs=LinkedList
[GridCacheMvccCandidate [nodeId=d38d1293-9dd5-4b4e-9934-97ba3fbafa62,
ver=GridCacheVersion [topVer=205532486, order=1594223109711, nodeOrder=4],
threadId=4722, id=56494472, topVer=AffinityTopologyVersion [topVer=83,
minorTopVer=0], reentry=null, otherNodeId=35c37b55-ec30-4457-b861-403bcfc20c12,
otherVer=GridCacheVersion [topVer=205532486, order=1594223109704,
nodeOrder=51], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null,
serOrder=null, key=KeyCacheObjectImpl [part=72, val=GridCacheInternalKeyImpl
[name=lt_642138, grpName=default-volatile-ds-group], hasValBytes=true],
masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=2]]], prepared=0,
locked=false, nodeId=null, locMapped=false, expiryPlc=null,
transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
xidVer=GridCacheVersion [topVer=205532486, order=1594223109711,
nodeOrder=4]]]], flags=onePhase|last|sys, super=GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=205532486, order=1594223109704, nodeOrder=51],
committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage
[cacheId=0, super=GridCacheMessage [msgId=11018564, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=81, minorTopVer=44],
err=null, skipPrepare=false]]]]]]
class org.apache.ignite.IgniteException: Failed to resolve nodes topology
[cacheGrp=N/A, topVer=AffinityTopologyVersion [topVer=83, minorTopVer=0],
history=[AffinityTopologyVersion [topVer=84, minorTopVer=0],
AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion
[topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0],
AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion
[topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0],
AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion
[topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93,
minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93,
minorTopVer=0]], locNode=TcpDiscoveryNode
[id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62,
consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList
[192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4,
intOrder=3, lastExchangeTime=1594054666300, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=false]]
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.resolveDiscoCache(GridDiscoveryManager.java:1999)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.node(GridDiscoveryManager.java:1821)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1056)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:411)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:577)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:374)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest0(IgniteTxHandler.java:183)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:161)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:123)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:205)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:203)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1847)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1472)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1367)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:565)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:745)
[18:44:06,665][WARNING][sys-stripe-3-#4][CacheDiagnosticManager] Page locks
dump:
Thread=[name=auth-#69, id=119], state=WAITING
Locked pages = []
Locked pages log: name=auth-#69 time=(1594233846390, 2020-07-08 18:44:06.390)
Thread=[name=checkpoint-runner-#113, id=167], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#113 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=checkpoint-runner-#114, id=168], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#114 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=checkpoint-runner-#115, id=169], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#115 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=checkpoint-runner-#116, id=170], state=RUNNABLE
Locked pages = []
Locked pages log: name=checkpoint-runner-#116 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=client-connector-#6594, id=6902], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6594 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=client-connector-#6595, id=6903], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6595 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=client-connector-#6598, id=6906], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6598 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=client-connector-#6599, id=6907], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6599 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=client-connector-#6602, id=6911], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6602 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=client-connector-#6603, id=6912], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6603 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=client-connector-#6606, id=6915], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6606 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=client-connector-#6607, id=6916], state=WAITING
Locked pages = []
Locked pages log: name=client-connector-#6607 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=db-checkpoint-thread-#112, id=166], state=WAITING
Locked pages = []
Locked pages log: name=db-checkpoint-thread-#112 time=(1594233846390,
2020-07-08 18:44:06.390)
Thread=[name=dms-writer-thread-#76, id=128], state=WAITING
Locked pages = []
Locked pages log: name=dms-writer-thread-#76 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=exchange-worker-#70, id=122], state=RUNNABLE
Locked pages = []
Locked pages log: name=exchange-worker-#70 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=main, id=1], state=WAITING
Locked pages = []
Locked pages log: name=main time=(1594233846390, 2020-07-08 18:44:06.390)
Thread=[name=sys-stripe-0-#1, id=20], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-0-#1 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-1-#2, id=21], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-1-#2 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-10-#11, id=30], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-10-#11 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-11-#12, id=31], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-11-#12 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-12-#13, id=32], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-12-#13 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-13-#14, id=33], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-13-#14 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-14-#15, id=34], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-14-#15 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-15-#16, id=35], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-15-#16 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-16-#17, id=36], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-16-#17 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-17-#18, id=37], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-17-#18 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-18-#19, id=38], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-18-#19 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-19-#20, id=39], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-19-#20 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-2-#3, id=22], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-2-#3 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-20-#21, id=40], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-20-#21 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-21-#22, id=41], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-21-#22 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-22-#23, id=42], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-22-#23 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-23-#24, id=43], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-23-#24 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-24-#25, id=44], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-24-#25 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-25-#26, id=45], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-25-#26 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-26-#27, id=46], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-26-#27 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-27-#28, id=47], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-27-#28 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-28-#29, id=48], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-28-#29 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-29-#30, id=49], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-29-#30 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-3-#4, id=23], state=RUNNABLE
Locked pages = []
Locked pages log: name=sys-stripe-3-#4 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-30-#31, id=50], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-30-#31 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-31-#32, id=51], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-31-#32 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-4-#5, id=24], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-4-#5 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-5-#6, id=25], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-5-#6 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-6-#7, id=26], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-6-#7 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-7-#8, id=27], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-7-#8 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-8-#9, id=28], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-8-#9 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=sys-stripe-9-#10, id=29], state=WAITING
Locked pages = []
Locked pages log: name=sys-stripe-9-#10 time=(1594233846390, 2020-07-08
18:44:06.390)
Thread=[name=ttl-cleanup-worker-#108, id=160], state=TIMED_WAITING
Locked pages = []
Locked pages log: name=ttl-cleanup-worker-#108 time=(1594233846390, 2020-07-08
18:44:06.390)
[18:44:06,666][SEVERE][sys-stripe-3-#4][] JVM will be halted immediately due to
the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION,
err=class o.a.i.IgniteException: Failed to resolve nodes topology
[cacheGrp=SQL_PUBLIC_MILLION, topVer=AffinityTopologyVersion [topVer=82,
minorTopVer=0], history=[AffinityTopologyVersion [topVer=84, minorTopVer=0],
AffinityTopologyVersion [topVer=85, minorTopVer=0], AffinityTopologyVersion
[topVer=86, minorTopVer=0], AffinityTopologyVersion [topVer=87, minorTopVer=0],
AffinityTopologyVersion [topVer=88, minorTopVer=0], AffinityTopologyVersion
[topVer=89, minorTopVer=0], AffinityTopologyVersion [topVer=90, minorTopVer=0],
AffinityTopologyVersion [topVer=91, minorTopVer=0], AffinityTopologyVersion
[topVer=92, minorTopVer=0], AffinityTopologyVersion [topVer=93,
minorTopVer=0]], snap=Snapshot [topVer=AffinityTopologyVersion [topVer=93,
minorTopVer=0]], locNode=TcpDiscoveryNode
[id=d38d1293-9dd5-4b4e-9934-97ba3fbafa62,
consistentId=19764808-a91c-4a7e-bfe4-e384e1ca19af, addrs=ArrayList
[192.168.1.6], sockAddrs=HashSet [/192.168.1.6:47500], discPort=47500, order=4,
intOrder=3, lastExchangeTime=1594054666300, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=false]]]]
