Andrey Kalinin created IGNITE-9954: -------------------------------------- Summary: An error in rolling back the transaction caused the cluster to hang Key: IGNITE-9954 URL: https://issues.apache.org/jira/browse/IGNITE-9954 Project: Ignite Issue Type: Bug Reporter: Andrey Kalinin
Starts a transaction with a small timeout on one node, we observe the rollback of the transaction by timeout: {code:java} 2018-10-18 02:14:13.339 [ERROR][sys-stripe-48-#49%GRID%GridNodeName%][o.a.i.i.p.c.t.IgniteTxHandler] Failed to process prepare request: GridDhtTxPrepareRequest [nearNodeId=ab0c3e9f-1642-4779-a6a0-dd10a6efd387, futId=54208048661-53dd6648-cc70-4076-a608-f6072645e609, miniId=1, topVer=AffinityTopologyVersion [topVer=338, minorTopVer=0], invalidateNearEntries={}, nearWrites=null, owned=null, nearXidVer=GridCacheVersion [topVer=151260338, order=1543126364337, nodeOrder=172], subjId=ab0c3e9f-1642-4779-a6a0-dd10a6efd387, taskNameHash=0, preloadKeys=null, skipCompletedVers=false, super=GridDistributedTxPrepareRequest [threadId=732, concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, writeVer=GridCacheVersion [topVer=151260338, order=1543126439977, nodeOrder=81], timeout=9, reads=null, writes=ArrayList [IgniteTxEntry [key=KeyCacheObjectImpl [part=12715, val=ids_name, hasValBytes=true], cacheId=-1934881220, txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=12715, val=ids_name, hasValBytes=true], cacheId=-1934881220], val=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=], prevVal=[op=NOOP, val=null], oldVal=CacheObjectImpl [val=null, hasValBytes=true][op=UPDATE, val=], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=GridCacheMapEntry [key=KeyCacheObjectImpl [part=12715, val=ids_name, hasValBytes=true], val=null, startVer=1543128775655, ver=GridCacheVersion [topVer=151260338, order=1543126364674, nodeOrder=81], hash=-864500235, extras=GridCacheObsoleteEntryExtras [obsoleteVer=GridCacheVersion [topVer=2147483647, order=0, nodeOrder=0]], flags=2]GridDistributedCacheEntry [super=]GridDhtCacheEntry [rdrs=ReaderId[] [], part=12715, super=], prepared=0, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2, partUpdateCntr=0, serReadVer=null, xidVer=null]], dhtVers=null, txSize=0, plc=2, txState=null, flags=onePhase|last, super=GridDistributedBaseMessage [ver=GridCacheVersion [topVer=151260338, order=1543126439864, nodeOrder=81], committedVers=null, rolledbackVers=null, cnt=0, super=GridCacheIdMessage [cacheId=0]]]] org.apache.ignite.internal.transactions.IgniteTxTimeoutCheckedException: Transaction timed out: GridCacheSharedManagerAdapter [starting=true, stop=false] at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.prepareTx(IgniteTxManager.java:868) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.prepareRemoteTx(GridDistributedTxRemoteAdapter.java:406) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.startRemoteTx(IgniteTxHandler.java:1754) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:1121) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:101) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:205) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:203) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1058) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:583) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:382) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:308) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1$2$1.run(GridCacheIoManager.java:277) at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) {code} at the same time on another node: {code:java} 2018-10-18 02:14:03.948 [WARN ][sys-stripe-36-#37%GRID%GridNodeName%][o.apache.ignite.cache.msg.tx.finish] Received finish request for completed transaction (the message may be too late) [txId=GridCacheVersion [topVer=151260338, order=1543126364337, nodeOrder=172], dhtTxId=null, node=ab0c3e9f-1642-4779-a6a0-dd10a6efd387, commit=false] {code} transaction hangs {code:java} TcpDiscoveryNode [id=db0b2eb9-c3ed-465a-8799-23e8ff336ffd, addrs=[192.168.1.29], order=338, ver=2.5, isClient=false, consistentId=node3] Tx: [xid=8b7d0894761-00000000-0904-0cb2-0000-000000000051, label=null, state=MARKED_ROLLBACK, startTime=2018-10-18 02:14:13.307, duration=37507, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=338, minorTopVer=0], timeout=9, size=1, dhtNodes=N/A, nearXid=1b0bf794761-00000000-0904-0cb2-0000-0000000000ac, parentNodeIds=[ab0c3e9f, 4e2505b4]] {code} exchange did not complete when changed the topology no further transactions take place {code:java} TcpDiscoveryNode [id=4f06e4cb-5a79-4ce9-b5d8-e53c164e1791, addrs=[192.168.1.35], order=337, ver=2.5, isClient=false, consistentId=node6] Tx: [xid=4052cbfb761-00000000-0904-0cb5-0000-000000000151, label=null, state=ACTIVE, startTime=2018-10-18 05:34:33.358, duration=25487, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, topVer=N/A, timeout=0, size=0, dhtNodes=[], nearXid=4052cbfb761-00000000-0904-0cb5-0000-000000000151, parentNodeIds=[4f06e4cb]] {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)