Hi,
I have apache ignite cluster of 3 ignite server and more than 20 ignite
thin clients (each thin client being on separate VM). These thin clients
are trying to create caches at approximately the same time parallely and
also starting with cache CRUD operations after that.
Looks like partition map exchange process and cache CRUD operations in
parallel are causing deadlock or lock acquire failures.
*What should be the strategy to handle this scenario ?*
Ignite server has below errors:
*Exception stack trace 1:*
WARNING: Dumping the near node thread that started transaction
[xidVer=GridCacheVersion [topVer=247332659, order=1635852705217,
nodeOrder=1], nodeId=2735bef0-7404-41e3-843f-7043490c9d84]
Stack trace of the transaction owner thread:
Thread [name="client-connector-#56%perf-dn1%", id=93, state=WAITING,
blockCnt=5023, waitCnt=36165]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
at
o.a.i.i.processors.cache.GridCacheAdapter$41.op(GridCacheAdapter.java:3430)
at
o.a.i.i.processors.cache.GridCacheAdapter$41.op(GridCacheAdapter.java:3423)
at
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4480)
at
o.a.i.i.processors.cache.GridCacheAdapter.remove0(GridCacheAdapter.java:3423)
at
o.a.i.i.processors.cache.GridCacheAdapter.remove(GridCacheAdapter.java:3405)
at
o.a.i.i.processors.cache.GridCacheAdapter.remove(GridCacheAdapter.java:3388)
at
o.a.i.i.processors.cache.IgniteCacheProxyImpl.remove(IgniteCacheProxyImpl.java:1438)
at
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.remove(GatewayProtectedCacheProxy.java:964)
at
o.a.i.i.processors.platform.client.cache.ClientCacheRemoveKeyRequest.process(ClientCacheRemoveKeyRequest.java:41)
at
o.a.i.i.processors.platform.client.ClientRequestHandler.handle(ClientRequestHandler.java:77)
at
o.a.i.i.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:204)
at
o.a.i.i.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:55)
at
o.a.i.i.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
at
o.a.i.i.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
o.a.i.i.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at o.a.i.i.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
*Exception stack trace 2:*
WARNING: >>> Transaction [startTime=11:39:27.214,
curTime=11:40:36.277, systemTime=0, userTime=69063, tx=GridNearTxLocal
[mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false,
colocatedLocallyMapped=false, needCheckBackup=null,
hasRemoteLocks=false, trackTimeout=false, systemTime=44700,
systemStartTime=0, prepareStartTime=0, prepareTime=0,
commitOrRollbackStartTime=0, commitOrRollbackTime=0, lb=null,
mvccOp=null, qryId=-1, crdVer=0,
thread=client-connector-#57%perf-dn1%, mappings=IgniteTxMappingsImpl
[], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false,
span=o.a.i.i.processors.tracing.NoopSpan@4a931268,
nearNodes=KeySetView [], dhtNodes=KeySetView [], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null,
sndTransformedVals=false, depEnabled=false, txState=IgniteTxStateImpl
[activeCacheIds=[], recovery=null, mvccEnabled=null,
mvccCachingCacheIds=[], txMap=EmptySet []], super=IgniteTxAdapter
[xidVer=GridCacheVersion [topVer=247332659, order=1635852705226,
nodeOrder=1], writeVer=null, implicit=false, loc=true, threadId=95,
startTime=1635853167214, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, sys=false, plc=2, commitVer=null,
finalizing=NONE, invalidParts=null, state=SUSPENDED, timedOut=false,
topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0],
mvccSnapshot=null, skipCompletedVers=false, parentTx=null,
duration=69079ms, onePhaseCommit=false], size=0]]]]
Nov 2, 2021 11:40:36 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: First 10 long running cache futures [total=16]
Nov 2, 2021 11:40:36 AM org.apache.ignite.logger.java.JavaLogger warning
WARNING: >>> Future [startTime=11:39:27.324, curTime=11:40:36.277,
fut=GridDhtLockFuture
[span=o.a.i.i.processors.tracing.NoopSpan@4a931268,
nearNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
nearLockVer=GridCacheVersion [topVer=247332659, order=1635852705208,
nodeOrder=1], topVer=AffinityTopologyVersion [topVer=1,
minorTopVer=162], threadId=124,
futId=be58a60ec71-1d64903c-c700-4deb-bace-cc5158713120,
lockVer=GridCacheVersion [topVer=247332659, order=1635852705208,
nodeOrder=1], read=false, err=null, timedOut=false, timeout=0,
tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl []dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]], prepared=0,
locked=false, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=247332659, order=1635852705208, nodeOrder=1]]]],
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=247332659,
order=1635852705208, nodeOrder=1], writeVer=null, implicit=false,
loc=true, threadId=124, startTime=1635853167214,
nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, sys=false, plc=2, commitVer=null,
finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
mvccSnapshot=null, skipCompletedVers=false, parentTx=null,
duration=69094ms, onePhaseCommit=false], size=1]],
nearLocallyMapped=false, colocatedLocallyMapped=true,
needCheckBackup=null, hasRemoteLocks=false, trackTimeout=false,
systemTime=75000, systemStartTime=971108549857700, prepareStartTime=0,
prepareTime=0, commitOrRollbackStartTime=0, commitOrRollbackTime=0,
lb=null, mvccOp=null, qryId=-1, crdVer=0,
thread=client-connector-#84%perf-dn1%, mappings=IgniteTxMappingsImpl
[]dNearNodes=null, ownerVer=GridCacheVersion [topVer=247332659,
order=1635852705211, nodeOrder=1], serOrder=null,
key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]], prepared=0,
locked=false, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=247332659, order=1635852705208, nodeOrder=1]]]],
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=247332659,
order=1635852705208, nodeOrder=1], writeVer=null, implicit=false,
loc=true, threadId=124, startTime=1635853167214,
nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, sys=false, plc=2, commitVer=null,
finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
mvccSnapshot=null, skipCompletedVers=false, parentTx=null,
duration=69094ms, onePhaseCommit=false], size=1]],
super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false,
span=o.a.i.i.processors.tracing.NoopSpan@4a931268,
nearNodes=KeySetView [], dhtNodes=KeySetView [], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null,
sndTransformedVals=false, depEnabled=false, txState=IgniteTxStateImpl
[activeCacheIds=[585748697], recovery=false, mvccEnabled=false,
mvccCachingCacheIds=[], txMap=ArrayList [IgniteTxEntry
[txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true], cacheId=585748697], val=TxEntryValueHolder
[val=null, op=DELETE], prevVal=TxEntryValueHolder [val=null, op=NOOP],
oldVal=TxEntryValueHolder [val=null, op=NOOP],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1,
conflictVer=null, explicitVer=null, dhtVer=null,
filters=CacheEntryPredicate[] [], filtersPassed=false,
filtersSet=true, entry=GridDhtCacheEntry [rdrs=ReaderId[] [],
part=244, super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true], val=null, ver=GridCacheVersion [topVer=247332659,
order=1635852705229, nodeOrder=1], hash=1085684290,
extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc [locs=LinkedList
[GridCacheMvccCandidate [nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
ver=GridCacheVersion [topVer=247332659, order=1635852705207,
nodeOrder=1], threadId=122, id=2104, topVer=AffinityTopologyVersion
[topVer=1, minorTopVer=162], reentry=null,
otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705207,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null], GridCacheMvccCandidate
[nodeId=2735bef0-7404-41e3-843f-7043490c9d84, ver=GridCacheVersion
[topVer=247332659, order=1635852705208, nodeOrder=1], threadId=124,
id=2102, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
reentry=null, otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705208,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null], GridCacheMvccCandidate
[nodeId=2735bef0-7404-41e3-843f-7043490c9d84, ver=GridCacheVersion
[topVer=247332659, order=1635852705213, nodeOrder=1], threadId=122,
id=2120, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
reentry=null, otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705213,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705207,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null], GridCacheMvccCandidate
[nodeId=2735bef0-7404-41e3-843f-7043490c9d84, ver=GridCacheVersion
[topVer=247332659, order=1635852705214, nodeOrder=1], threadId=123,
id=2118, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
reentry=null, otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705214,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705207,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null], GridCacheMvccCandidate
[nodeId=2735bef0-7404-41e3-843f-7043490c9d84, ver=GridCacheVersion
[topVer=247332659, order=1635852705217, nodeOrder=1], threadId=93,
id=2108, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
reentry=null, otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705217,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null], GridCacheMvccCandidate
[nodeId=2735bef0-7404-41e3-843f-7043490c9d84, ver=GridCacheVersion
[topVer=247332659, order=1635852705218, nodeOrder=1], threadId=115,
id=2106, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
reentry=null, otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705218,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null], GridCacheMvccCandidate
[nodeId=2735bef0-7404-41e3-843f-7043490c9d84, ver=GridCacheVersion
[topVer=247332659, order=1635852705222, nodeOrder=1], threadId=95,
id=2110, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
reentry=null, otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705222,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null], GridCacheMvccCandidate
[nodeId=2735bef0-7404-41e3-843f-7043490c9d84, ver=GridCacheVersion
[topVer=247332659, order=1635852705223, nodeOrder=1], threadId=120,
id=2112, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
reentry=null, otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705223,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null], GridCacheMvccCandidate
[nodeId=2735bef0-7404-41e3-843f-7043490c9d84, ver=GridCacheVersion
[topVer=247332659, order=1635852705227, nodeOrder=1], threadId=118,
id=2114, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
reentry=null, otherNodeId=2735bef0-7404-41e3-843f-7043490c9d84,
otherVer=GridCacheVersion [topVer=247332659, order=1635852705227,
nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]dNearNodes=null, ownerVer=GridCacheVersion
[topVer=247332659, order=1635852705211, nodeOrder=1], serOrder=null,
key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]], prepared=0,
locked=false, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=247332659, order=1635852705208, nodeOrder=1]]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]], prepared=0,
locked=false, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=247332659, order=1635852705208,
nodeOrder=1]]]]dNearNodes=null, ownerVer=GridCacheVersion
[topVer=247332659, order=1635852705211, nodeOrder=1], serOrder=null,
key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]], prepared=0,
locked=false, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=247332659, order=1635852705208, nodeOrder=1]]]],
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=247332659,
order=1635852705208, nodeOrder=1], writeVer=null, implicit=false,
loc=true, threadId=124, startTime=1635853167214,
nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, sys=false, plc=2, commitVer=null,
finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
mvccSnapshot=null, skipCompletedVers=false, parentTx=null,
duration=69094ms, onePhaseCommit=false], size=1]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]], prepared=0,
locked=false, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=247332659, order=1635852705208, nodeOrder=1]]]],
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=247332659,
order=1635852705208, nodeOrder=1], writeVer=null, implicit=false,
loc=true, threadId=124, startTime=1635853167214,
nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, sys=false, plc=2, commitVer=null,
finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
mvccSnapshot=null, skipCompletedVers=false, parentTx=null,
duration=69094ms, onePhaseCommit=false], size=1]]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]], prepared=0,
locked=false, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=247332659, order=1635852705208, nodeOrder=1]]]],
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=247332659,
order=1635852705208, nodeOrder=1], writeVer=null, implicit=false,
loc=true, threadId=124, startTime=1635853167214,
nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, sys=false, plc=2, commitVer=null,
finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
mvccSnapshot=null, skipCompletedVers=false, parentTx=null,
duration=69094ms, onePhaseCommit=false], size=1]]]dNearNodes=null,
ownerVer=GridCacheVersion [topVer=247332659, order=1635852705211,
nodeOrder=1], serOrder=null, key=KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true],
masks=local=1|owner=0|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=3]]], prepared=0,
locked=false, nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2,
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=247332659, order=1635852705208, nodeOrder=1]]]],
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=247332659,
order=1635852705208, nodeOrder=1], writeVer=null, implicit=false,
loc=true, threadId=124, startTime=1635853167214,
nodeId=2735bef0-7404-41e3-843f-7043490c9d84,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, sys=false, plc=2, commitVer=null,
finalizing=NONE, invalidParts=null, state=ACTIVE, timedOut=false,
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=162],
mvccSnapshot=null, skipCompletedVers=false, parentTx=null,
duration=69094ms, onePhaseCommit=false], size=1]]], mapped=false,
trackable=true, createTtl=-1, accessTtl=-1, needReturnVal=false,
skipStore=false, keepBinary=true, innerFuts=EmptyList [],
pendingLocks=HashSet [KeyCacheObjectImpl [part=244,
val=data=6ff0c60ec71-625345be-9a91-497a-895e-abbe5df9da3d],
hasValBytes=true]], super=GridCompoundIdentityFuture
[super=GridCompoundFuture [rdc=Bool reducer: true, initFlag=0,
lsnrCalls=0, done=false, cancelled=false, err=null, futs=EmptyList
[]]]]]
Nov 2, 2021 11:40:36 AM org.apache.ignite.logger.java.JavaLogger warning
*Exception stack trace 3 when TxTimeoutOnPartitionMapExchange is set to
some value*
Nov 2, 2021 1:11:19 PM org.apache.ignite.logger.java.JavaLogger warning
WARNING: The transaction was forcibly rolled back on partition map
exchange because a timeout is reached:
[tx=GridNearTxLocal[xid=e0e3db0ec71-00000000-0ebe-146c-0000-000000000001,
xidVersion=GridCacheVersion [topVer=247338092, order=1635858071054,
nodeOrder=1], nearXidVersion=GridCacheVersion [topVer=247338092,
order=1635858071054, nodeOrder=1], concurrency=PESSIMISTIC,
isolation=REPEATABLE_READ, state=ACTIVE, invalidate=false,
rollbackOnly=false, nodeId=1ed7077a-5547-41c1-94ab-d246944ed4a9,
timeout=0, startTime=1635858659324, duration=20140, label=null],
topVer=AffinityTopologyVersion [topVer=1, minorTopVer=104]]
Nov 2, 2021 1:11:19 PM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to process client request
[req=o.a.i.i.processors.platform.client.cache.ClientCacheRemoveKeyRequest@2fea4a3]
javax.cache.CacheException: class
org.apache.ignite.transactions.TransactionRollbackException: Failed to
finish transaction because it has been rolled back [timeout=0,
tx=GridNearTxLocal[xid=f0e3db0ec71-00000000-0ebe-146c-0000-000000000001,
xidVersion=GridCacheVersion [topVer=247338092, order=1635858071055,
nodeOrder=1], nearXidVersion=GridCacheVersion [topVer=247338092,
order=1635858071055, nodeOrder=1], concurrency=PESSIMISTIC,
isolation=REPEATABLE_READ, state=ACTIVE, invalidate=false,
rollbackOnly=false, nodeId=1ed7077a-5547-41c1-94ab-d246944ed4a9,
timeout=0, startTime=1635858659324, duration=20140, label=null]]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1266)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2084)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.remove(IgniteCacheProxyImpl.java:1441)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.remove(GatewayProtectedCacheProxy.java:964)
at
org.apache.ignite.internal.processors.platform.client.cache.ClientCacheRemoveKeyRequest.process(ClientCacheRemoveKeyRequest.java:41)
at
org.apache.ignite.internal.processors.platform.client.ClientRequestHandler.handle(ClientRequestHandler.java:77)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:204)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:55)
at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class
org.apache.ignite.transactions.TransactionRollbackException: Failed to
finish transaction because it has been rolled back [timeout=0,
tx=GridNearTxLocal[xid=f0e3db0ec71-00000000-0ebe-146c-0000-000000000001,
xidVersion=GridCacheVersion [topVer=247338092, order=1635858071055,
nodeOrder=1], nearXidVersion=GridCacheVersion [topVer=247338092,
order=1635858071055, nodeOrder=1], concurrency=PESSIMISTIC,
isolation=REPEATABLE_READ, state=ACTIVE, invalidate=false,
rollbackOnly=false, nodeId=1ed7077a-5547-41c1-94ab-d246944ed4a9,
timeout=0, startTime=1635858659324, duration=20140, label=null]]
at
org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:974)
at
org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:972)
... 16 more
--
Sumit Deshinge
R&D Engineer | Symantec Enterprise Division
Broadcom Software
Email: Sumit Deshinge <sumit.deshi...@broadcom.com>
--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.
