[ https://issues.apache.org/jira/browse/IGNITE-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey Uttsel updated IGNITE-18737: ----------------------------------- Description: h3. *Motivation* After https://issues.apache.org/jira/browse/IGNITE-18088 test from ItIgniteNodeRestartTest started to fail. Need to investigate the reason of fails {noformat} Caused by: java.util.concurrent.CancellationException at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:491) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:473) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.refreshAndGetLeaderWithTerm(RaftGroupServiceImpl.java:232) at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.ensureReplicaIsPrimary(PartitionReplicaListener.java:1880) at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.invoke(PartitionReplicaListener.java:275) at org.apache.ignite.internal.replicator.Replica.processRequest(Replica.java:61) at org.apache.ignite.internal.replicator.ReplicaManager.lambda$new$5(ReplicaManager.java:162) at org.apache.ignite.network.DefaultMessagingService.sendToSelf(DefaultMessagingService.java:279) at org.apache.ignite.network.DefaultMessagingService.invoke0(DefaultMessagingService.java:226) at org.apache.ignite.network.DefaultMessagingService.invoke(DefaultMessagingService.java:154) at org.apache.ignite.internal.replicator.ReplicaService.sendToReplica(ReplicaService.java:91) at org.apache.ignite.internal.replicator.ReplicaService.invoke(ReplicaService.java:178) at org.apache.ignite.internal.tx.impl.TxManagerImpl.cleanup(TxManagerImpl.java:150) at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processTxFinishAction$42(PartitionReplicaListener.java:984) at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$38(RaftGroupServiceImpl.java:526) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:479) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183){noformat} The root cause is that distribution zones data nodes watch listener in TableManager invokes updatePendingAssignmentsKeys with the same data nodes as in table configuration assignments so RaftGroupServiceImpl stops and new the same RaftGroupServiceImpl starts. It cancels invokes to old RaftGroupServiceImpl so we have CancellationException at sendWithRetry. h3. *Implementation Notes* It is not need to change the assignments if the calculated assignments equals to the assignments in the table configuration. was: After https://issues.apache.org/jira/browse/IGNITE-18088 test from ItIgniteNodeRestartTest started to fail. Need to investigate the reason of fails https://ci.ignite.apache.org/test/-8438821239346742957?currentProjectId=ApacheIgnite3xGradle_Test_IntegrationTests&expandTestHistoryChartSection=true&branch=%3Cdefault%3E {noformat} org.apache.ignite.lang.IgniteInternalException: IGN-CMN-65535 TraceId:9b2c2e61-1676-4dc3-8d76-2494803d7a2b No such partition 0 in table TABLE1 at org.apache.ignite.internal.table.distributed.storage.InternalTableImpl.partitionRaftGroupService(InternalTableImpl.java:1124) at org.apache.ignite.internal.table.distributed.TableManager.lambda$createPartitionMover$79(TableManager.java:2131) at org.apache.ignite.internal.table.distributed.PartitionMover.movePartition(PartitionMover.java:66) at org.apache.ignite.internal.table.distributed.raft.RebalanceRaftGroupEventsListener.lambda$onLeaderElected$0(RebalanceRaftGroupEventsListener.java:202) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) {noformat} > Tests from ItIgniteNodeRestartTest became flaky > -------------------------------------------------- > > Key: IGNITE-18737 > URL: https://issues.apache.org/jira/browse/IGNITE-18737 > Project: Ignite > Issue Type: Bug > Reporter: Mirza Aliev > Assignee: Sergey Uttsel > Priority: Major > Labels: ignite-3 > Time Spent: 1.5h > Remaining Estimate: 0h > > h3. *Motivation* > After https://issues.apache.org/jira/browse/IGNITE-18088 test from > ItIgniteNodeRestartTest started to fail. Need to investigate the reason of > fails > {noformat} > Caused by: java.util.concurrent.CancellationException > at > java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396) > at > org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:491) > at > org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:473) > at > org.apache.ignite.internal.raft.RaftGroupServiceImpl.refreshAndGetLeaderWithTerm(RaftGroupServiceImpl.java:232) > at > org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.ensureReplicaIsPrimary(PartitionReplicaListener.java:1880) > at > org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.invoke(PartitionReplicaListener.java:275) > at > org.apache.ignite.internal.replicator.Replica.processRequest(Replica.java:61) > at > org.apache.ignite.internal.replicator.ReplicaManager.lambda$new$5(ReplicaManager.java:162) > at > org.apache.ignite.network.DefaultMessagingService.sendToSelf(DefaultMessagingService.java:279) > at > org.apache.ignite.network.DefaultMessagingService.invoke0(DefaultMessagingService.java:226) > at > org.apache.ignite.network.DefaultMessagingService.invoke(DefaultMessagingService.java:154) > at > org.apache.ignite.internal.replicator.ReplicaService.sendToReplica(ReplicaService.java:91) > at > org.apache.ignite.internal.replicator.ReplicaService.invoke(ReplicaService.java:178) > at > org.apache.ignite.internal.tx.impl.TxManagerImpl.cleanup(TxManagerImpl.java:150) > at > org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processTxFinishAction$42(PartitionReplicaListener.java:984) > at > java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) > at > java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) > at > java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) > at > org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$38(RaftGroupServiceImpl.java:526) > at > java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) > at > java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:479) > at > java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) > at > java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) > at > java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) > at > java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) > at > java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183){noformat} > The root cause is that distribution zones data nodes watch listener in > TableManager invokes updatePendingAssignmentsKeys with the same data nodes as > in table configuration assignments so RaftGroupServiceImpl stops and new the > same RaftGroupServiceImpl starts. It cancels invokes to old > RaftGroupServiceImpl so we have CancellationException at sendWithRetry. > h3. *Implementation Notes* > It is not need to change the assignments if the calculated assignments equals > to the assignments in the table configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)