alex-plekhanov commented on code in PR #13339: URL: https://github.com/apache/ignite/pull/13339#discussion_r3688847113
########## docs/_docs/key-value-api/transactions.adoc: ########## @@ -322,6 +341,89 @@ A transaction might fail with the following exceptions: == Long Running Transactions Termination +The `tx` command of the `control.sh` utility allows you to search for transactions based on various criteria (such as ID, duration, or label) and roll back transactions. Since transactions in Apache Ignite are generally distributed (multiple nodes are involved in processing a single transaction), this command may display the same transaction as present on multiple nodes simultaneously. + +Rolling back a transaction using the `tx` command is only possible for the `ACTIVE`, `PREPARING`, and `PREPARED` statuses. A transaction in statuses related to the `rollback` procedure is already in the process of rolling back. Transactions in the `COMMITTING` and `COMMITTED` statuses cannot be rolled back due to 2PC protocol restrictions. + +* First termination stage — cancelling transactions using `tx kill`. To cancel transactions, use the `control.sh` utility to minimize the impact on the cluster: +. Execute the `control.sh --tx --min-duration <transaction_timeout_in_ms>` command in the cluster. Example output: Review Comment: `--min-duration` - in seconds, not ms. ########## docs/_docs/key-value-api/transactions.adoc: ########## @@ -322,6 +341,89 @@ A transaction might fail with the following exceptions: == Long Running Transactions Termination +The `tx` command of the `control.sh` utility allows you to search for transactions based on various criteria (such as ID, duration, or label) and roll back transactions. Since transactions in Apache Ignite are generally distributed (multiple nodes are involved in processing a single transaction), this command may display the same transaction as present on multiple nodes simultaneously. + +Rolling back a transaction using the `tx` command is only possible for the `ACTIVE`, `PREPARING`, and `PREPARED` statuses. A transaction in statuses related to the `rollback` procedure is already in the process of rolling back. Transactions in the `COMMITTING` and `COMMITTED` statuses cannot be rolled back due to 2PC protocol restrictions. Review Comment: I'm not sure we can rollback tx in `PREPARED` state. Can you please recheck it? ########## docs/_docs/key-value-api/transactions.adoc: ########## @@ -322,6 +341,89 @@ A transaction might fail with the following exceptions: == Long Running Transactions Termination +The `tx` command of the `control.sh` utility allows you to search for transactions based on various criteria (such as ID, duration, or label) and roll back transactions. Since transactions in Apache Ignite are generally distributed (multiple nodes are involved in processing a single transaction), this command may display the same transaction as present on multiple nodes simultaneously. + +Rolling back a transaction using the `tx` command is only possible for the `ACTIVE`, `PREPARING`, and `PREPARED` statuses. A transaction in statuses related to the `rollback` procedure is already in the process of rolling back. Transactions in the `COMMITTING` and `COMMITTED` statuses cannot be rolled back due to 2PC protocol restrictions. + +* First termination stage — cancelling transactions using `tx kill`. To cancel transactions, use the `control.sh` utility to minimize the impact on the cluster: +. Execute the `control.sh --tx --min-duration <transaction_timeout_in_ms>` command in the cluster. Example output: ++ +---- +Command [TX] started +Arguments: --tx --min-duration 30 +-------------------------------------------------------------------------------- +Matching transactions: +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=1, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest0] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +Command [TX] finished with code: 0 +---- +. Extract the `xid` from all transactions across all nodes and remove duplicate `xid` values. Duplicates occur because the same transaction can exist on multiple nodes — in this case, its `xid` will appear in the command output multiple times. +. For each `xid`, run the `tx kill` command to roll back that transaction. Example call: `control.sh --tx --xid <id> --kill`. Review Comment: `--tx -kill` can be used with `--min-duration`. Why we recomend to use kill txs one by one? ########## docs/_docs/key-value-api/transactions.adoc: ########## @@ -322,6 +341,89 @@ A transaction might fail with the following exceptions: == Long Running Transactions Termination +The `tx` command of the `control.sh` utility allows you to search for transactions based on various criteria (such as ID, duration, or label) and roll back transactions. Since transactions in Apache Ignite are generally distributed (multiple nodes are involved in processing a single transaction), this command may display the same transaction as present on multiple nodes simultaneously. + +Rolling back a transaction using the `tx` command is only possible for the `ACTIVE`, `PREPARING`, and `PREPARED` statuses. A transaction in statuses related to the `rollback` procedure is already in the process of rolling back. Transactions in the `COMMITTING` and `COMMITTED` statuses cannot be rolled back due to 2PC protocol restrictions. + +* First termination stage — cancelling transactions using `tx kill`. To cancel transactions, use the `control.sh` utility to minimize the impact on the cluster: +. Execute the `control.sh --tx --min-duration <transaction_timeout_in_ms>` command in the cluster. Example output: ++ +---- +Command [TX] started +Arguments: --tx --min-duration 30 +-------------------------------------------------------------------------------- +Matching transactions: +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=1, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest0] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +Command [TX] finished with code: 0 +---- +. Extract the `xid` from all transactions across all nodes and remove duplicate `xid` values. Duplicates occur because the same transaction can exist on multiple nodes — in this case, its `xid` will appear in the command output multiple times. +. For each `xid`, run the `tx kill` command to roll back that transaction. Example call: `control.sh --tx --xid <id> --kill`. + +* Second termination stage (optional) — restarting nodes that block transaction completion. Check if the first stage was sufficient to handle LRT: Review Comment: It's a strange recomendation. At least it's not a normal workflow and can't be recomended as `second stage`, it's an exceptional situation. ########## docs/_docs/key-value-api/transactions.adoc: ########## @@ -322,6 +341,89 @@ A transaction might fail with the following exceptions: == Long Running Transactions Termination +The `tx` command of the `control.sh` utility allows you to search for transactions based on various criteria (such as ID, duration, or label) and roll back transactions. Since transactions in Apache Ignite are generally distributed (multiple nodes are involved in processing a single transaction), this command may display the same transaction as present on multiple nodes simultaneously. + +Rolling back a transaction using the `tx` command is only possible for the `ACTIVE`, `PREPARING`, and `PREPARED` statuses. A transaction in statuses related to the `rollback` procedure is already in the process of rolling back. Transactions in the `COMMITTING` and `COMMITTED` statuses cannot be rolled back due to 2PC protocol restrictions. + +* First termination stage — cancelling transactions using `tx kill`. To cancel transactions, use the `control.sh` utility to minimize the impact on the cluster: +. Execute the `control.sh --tx --min-duration <transaction_timeout_in_ms>` command in the cluster. Example output: ++ +---- +Command [TX] started +Arguments: --tx --min-duration 30 +-------------------------------------------------------------------------------- +Matching transactions: +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=1, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest0] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +Command [TX] finished with code: 0 +---- +. Extract the `xid` from all transactions across all nodes and remove duplicate `xid` values. Duplicates occur because the same transaction can exist on multiple nodes — in this case, its `xid` will appear in the command output multiple times. +. For each `xid`, run the `tx kill` command to roll back that transaction. Example call: `control.sh --tx --xid <id> --kill`. + +* Second termination stage (optional) — restarting nodes that block transaction completion. Check if the first stage was sufficient to handle LRT: +. After completing the first stage, wait for a period equal to the configured transaction timeout. +. Run the `control.sh --tx --min-duration <transaction_timeout>` command. +If the list is empty, all LRTs have been successfully canceled, and the LRT impact should be resolved. +If the list is not empty, the command output will contain information about nodes and the transactions running on them: ++ +---- +Command [TX] started +Arguments: --tx +-------------------------------------------------------------------------------- +Matching transactions: +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=1, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest0] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] + Tx: [xid=<id>, label=label1, state=ACTIVE, startTime=YYYY-MM-DD 16:58:48.451, duration=0 sec, isolation=READ_COMMITTED, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=2147483 sec, size=111, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=2, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest1] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=1, dhtNodes=[<id>], nearXid=<id>, parentNodeIds=[<id>]] +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=3, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=true, consistentId=client] + Tx: [xid=<id>, label=label2, state=PREPARING, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=11, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +Command [TX] finished with code: 0 +---- +. For each transaction, extract the value of the `dhtNodes` field. +. For each ID in `dhtNodes`, obtain detailed node information using the `control.sh --system-view NODES` command. +. In the command output, find the node whose `nodeId` starts with the string from `dhtNodes`. +. Restart this node to unblock the transaction. + +== Causes of LRT +* High system load, which leads to slower transaction processing. Indicators: Review Comment: ** High cpu or disk utilization ** Increased latency of cache operation ? ########## docs/_docs/key-value-api/transactions.adoc: ########## @@ -322,6 +341,89 @@ A transaction might fail with the following exceptions: == Long Running Transactions Termination +The `tx` command of the `control.sh` utility allows you to search for transactions based on various criteria (such as ID, duration, or label) and roll back transactions. Since transactions in Apache Ignite are generally distributed (multiple nodes are involved in processing a single transaction), this command may display the same transaction as present on multiple nodes simultaneously. + +Rolling back a transaction using the `tx` command is only possible for the `ACTIVE`, `PREPARING`, and `PREPARED` statuses. A transaction in statuses related to the `rollback` procedure is already in the process of rolling back. Transactions in the `COMMITTING` and `COMMITTED` statuses cannot be rolled back due to 2PC protocol restrictions. + +* First termination stage — cancelling transactions using `tx kill`. To cancel transactions, use the `control.sh` utility to minimize the impact on the cluster: +. Execute the `control.sh --tx --min-duration <transaction_timeout_in_ms>` command in the cluster. Example output: ++ +---- +Command [TX] started +Arguments: --tx --min-duration 30 +-------------------------------------------------------------------------------- +Matching transactions: +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=1, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest0] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +Command [TX] finished with code: 0 +---- +. Extract the `xid` from all transactions across all nodes and remove duplicate `xid` values. Duplicates occur because the same transaction can exist on multiple nodes — in this case, its `xid` will appear in the command output multiple times. +. For each `xid`, run the `tx kill` command to roll back that transaction. Example call: `control.sh --tx --xid <id> --kill`. + +* Second termination stage (optional) — restarting nodes that block transaction completion. Check if the first stage was sufficient to handle LRT: +. After completing the first stage, wait for a period equal to the configured transaction timeout. +. Run the `control.sh --tx --min-duration <transaction_timeout>` command. +If the list is empty, all LRTs have been successfully canceled, and the LRT impact should be resolved. +If the list is not empty, the command output will contain information about nodes and the transactions running on them: ++ +---- +Command [TX] started +Arguments: --tx +-------------------------------------------------------------------------------- +Matching transactions: +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=1, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest0] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] + Tx: [xid=<id>, label=label1, state=ACTIVE, startTime=YYYY-MM-DD 16:58:48.451, duration=0 sec, isolation=READ_COMMITTED, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=2147483 sec, size=111, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=2, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest1] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=1, dhtNodes=[<id>], nearXid=<id>, parentNodeIds=[<id>]] +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=3, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=true, consistentId=client] + Tx: [xid=<id>, label=label2, state=PREPARING, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=11, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +Command [TX] finished with code: 0 +---- +. For each transaction, extract the value of the `dhtNodes` field. +. For each ID in `dhtNodes`, obtain detailed node information using the `control.sh --system-view NODES` command. +. In the command output, find the node whose `nodeId` starts with the string from `dhtNodes`. +. Restart this node to unblock the transaction. + +== Causes of LRT +* High system load, which leads to slower transaction processing. Indicators: +** log messages about the start of page eviction for in-memory data regions or page replacement for persistent data regions; +** growth of queues in the striped pool; +** increased checkpoint duration for clusters with persistence enabled; +** long GC pauses. +* Unstable network operation. Indicators: log messages about connection loss between nodes, socket closures, or network timeouts triggering. +* Resource-intensive operations (for example, deleting a cache that is part of a cache group). Indicators: queue growth on individual threads of the system pool and striped pool, while other threads and overall node utilization may remain low. +* Internal Apache Ignite bugs that can lead to deadlocks. Indicators: thread dumps (`thread dump`) show threads waiting to acquire a read lock (`readLock`) or write lock (`writeLock`) during checkpoint creation; over time, the thread state does not change (stack trace remains unchanged). Review Comment: It's strange to bind deadlocks only to checkpoint. ########## docs/_docs/key-value-api/transactions.adoc: ########## @@ -322,6 +341,89 @@ A transaction might fail with the following exceptions: == Long Running Transactions Termination +The `tx` command of the `control.sh` utility allows you to search for transactions based on various criteria (such as ID, duration, or label) and roll back transactions. Since transactions in Apache Ignite are generally distributed (multiple nodes are involved in processing a single transaction), this command may display the same transaction as present on multiple nodes simultaneously. + +Rolling back a transaction using the `tx` command is only possible for the `ACTIVE`, `PREPARING`, and `PREPARED` statuses. A transaction in statuses related to the `rollback` procedure is already in the process of rolling back. Transactions in the `COMMITTING` and `COMMITTED` statuses cannot be rolled back due to 2PC protocol restrictions. + +* First termination stage — cancelling transactions using `tx kill`. To cancel transactions, use the `control.sh` utility to minimize the impact on the cluster: +. Execute the `control.sh --tx --min-duration <transaction_timeout_in_ms>` command in the cluster. Example output: ++ +---- +Command [TX] started +Arguments: --tx --min-duration 30 +-------------------------------------------------------------------------------- +Matching transactions: +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=1, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest0] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 15:15:16.852, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +Command [TX] finished with code: 0 +---- +. Extract the `xid` from all transactions across all nodes and remove duplicate `xid` values. Duplicates occur because the same transaction can exist on multiple nodes — in this case, its `xid` will appear in the command output multiple times. +. For each `xid`, run the `tx kill` command to roll back that transaction. Example call: `control.sh --tx --xid <id> --kill`. + +* Second termination stage (optional) — restarting nodes that block transaction completion. Check if the first stage was sufficient to handle LRT: +. After completing the first stage, wait for a period equal to the configured transaction timeout. +. Run the `control.sh --tx --min-duration <transaction_timeout>` command. +If the list is empty, all LRTs have been successfully canceled, and the LRT impact should be resolved. +If the list is not empty, the command output will contain information about nodes and the transactions running on them: ++ +---- +Command [TX] started +Arguments: --tx +-------------------------------------------------------------------------------- +Matching transactions: +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=1, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest0] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=100, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] + Tx: [xid=<id>, label=label1, state=ACTIVE, startTime=YYYY-MM-DD 16:58:48.451, duration=0 sec, isolation=READ_COMMITTED, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=2147483 sec, size=111, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=2, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=false, consistentId=gridCommandHandlerTest1] + Tx: [xid=<id>, label=null, state=ACTIVE, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=1, dhtNodes=[<id>], nearXid=<id>, parentNodeIds=[<id>]] +TcpDiscoveryNode [id=<id>, addrs=[<address>], order=3, ver=2.18.0#YYYYMMDD-sha1:00000000, isClient=true, consistentId=client] + Tx: [xid=<id>, label=label2, state=PREPARING, startTime=YYYY-MM-DD 16:58:45.348, duration=3 sec, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=2], timeout=0 sec, size=11, dhtNodes=[<id>, <id>], nearXid=<id>, parentNodeIds=[<id>]] +Command [TX] finished with code: 0 +---- +. For each transaction, extract the value of the `dhtNodes` field. +. For each ID in `dhtNodes`, obtain detailed node information using the `control.sh --system-view NODES` command. +. In the command output, find the node whose `nodeId` starts with the string from `dhtNodes`. +. Restart this node to unblock the transaction. + +== Causes of LRT +* High system load, which leads to slower transaction processing. Indicators: +** log messages about the start of page eviction for in-memory data regions or page replacement for persistent data regions; +** growth of queues in the striped pool; +** increased checkpoint duration for clusters with persistence enabled; +** long GC pauses. +* Unstable network operation. Indicators: log messages about connection loss between nodes, socket closures, or network timeouts triggering. +* Resource-intensive operations (for example, deleting a cache that is part of a cache group). Indicators: queue growth on individual threads of the system pool and striped pool, while other threads and overall node utilization may remain low. Review Comment: snapshots, index rebuild, rebalancing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
