Nikita Sivkov created IGNITE-22005: -------------------------------------- Summary: 'Failed to process replica request' error under load with balance transfer scenario Key: IGNITE-22005 URL: https://issues.apache.org/jira/browse/IGNITE-22005 Project: Ignite Issue Type: Bug Affects Versions: 3.0.0-beta2 Environment: Cluster of 3 nodes Reporter: Nikita Sivkov Attachments: transfer_ign3.yaml
*Steps to reproduce:* Perform a long (about 2 hours) load test with a balance transfer scenario (see scenario pseudo code in attachments). *Expected result:* No errors happen. *Actual result:* Get error in server logs - {{Failed to process delayed response}} {code:java} 2024-04-05 17:50:50:776 +0300 [WARNING][%poc-tester-SERVER-192.168.1.97-id-0%JRaft-Request-Processor-1][NodeImpl] Node <27_part_23/poc-tester-SERVER-192.168.1.97-id-0> is not in active state, currTerm=2. 2024-04-05 17:50:50:778 +0300 [WARNING][%poc-tester-SERVER-192.168.1.97-id-0%Raft-Group-Client-5][ReplicaManager] Failed to process delayed response [request=ReadWriteSingleRowReplicaRequestImpl [commitPartitionId=TablePartitionIdMessageImpl [partitionId=21, tableId=123], coordinatorId=3de6f999-7ab9-4405-aff0-ee0c7e4886ce, enlistmentConsistencyToken=112218720633356321, full=false, groupId=123_part_21, requestType=RW_UPSERT, schemaVersion=1, timestampLong=112219169796915211, transactionId=018eaebd-88ba-0001-606d-622500000001]] java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:550) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$handleErrorResponse$44(RaftGroupServiceImpl.java:653) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.util.concurrent.TimeoutException ... 8 more 2024-04-05 17:50:50:780 +0300 [WARNING][%poc-tester-SERVER-192.168.1.97-id-0%JRaft-Request-Processor-27][NodeImpl] Node <99_part_6/poc-tester-SERVER-192.168.1.97-id-0> is not in active state, currTerm=3. {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)