[
https://issues.apache.org/jira/browse/IGNITE-18737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Uttsel updated IGNITE-18737:
-----------------------------------
Description:
h3. *Motivation*
After https://issues.apache.org/jira/browse/IGNITE-18088 test from
ItIgniteNodeRestartTest started to fail. Need to investigate the reason of fails
{noformat}
Caused by: java.util.concurrent.CancellationException
at
java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:491)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:473)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.refreshAndGetLeaderWithTerm(RaftGroupServiceImpl.java:232)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.ensureReplicaIsPrimary(PartitionReplicaListener.java:1880)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.invoke(PartitionReplicaListener.java:275)
at
org.apache.ignite.internal.replicator.Replica.processRequest(Replica.java:61)
at
org.apache.ignite.internal.replicator.ReplicaManager.lambda$new$5(ReplicaManager.java:162)
at
org.apache.ignite.network.DefaultMessagingService.sendToSelf(DefaultMessagingService.java:279)
at
org.apache.ignite.network.DefaultMessagingService.invoke0(DefaultMessagingService.java:226)
at
org.apache.ignite.network.DefaultMessagingService.invoke(DefaultMessagingService.java:154)
at
org.apache.ignite.internal.replicator.ReplicaService.sendToReplica(ReplicaService.java:91)
at
org.apache.ignite.internal.replicator.ReplicaService.invoke(ReplicaService.java:178)
at
org.apache.ignite.internal.tx.impl.TxManagerImpl.cleanup(TxManagerImpl.java:150)
at
org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processTxFinishAction$42(PartitionReplicaListener.java:984)
at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$38(RaftGroupServiceImpl.java:526)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:479)
at
java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at
java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at
java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at
java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at
java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183){noformat}
The root cause is that distribution zones data nodes watch listener in
TableManager invokes updatePendingAssignmentsKeys with the same data nodes as
in table configuration assignments so RaftGroupServiceImpl stops and new the
same RaftGroupServiceImpl starts. It cancels invokes to old
RaftGroupServiceImpl so we have CancellationException at sendWithRetry.
h3. *Implementation Notes*
It is not need to change the assignments if the calculated assignments equals
to the assignments in the table configuration.
was:
After https://issues.apache.org/jira/browse/IGNITE-18088 test from
ItIgniteNodeRestartTest started to fail. Need to investigate the reason of fails
https://ci.ignite.apache.org/test/-8438821239346742957?currentProjectId=ApacheIgnite3xGradle_Test_IntegrationTests&expandTestHistoryChartSection=true&branch=%3Cdefault%3E
{noformat}
org.apache.ignite.lang.IgniteInternalException: IGN-CMN-65535
TraceId:9b2c2e61-1676-4dc3-8d76-2494803d7a2b No such partition 0 in table TABLE1
at
org.apache.ignite.internal.table.distributed.storage.InternalTableImpl.partitionRaftGroupService(InternalTableImpl.java:1124)
at
org.apache.ignite.internal.table.distributed.TableManager.lambda$createPartitionMover$79(TableManager.java:2131)
at
org.apache.ignite.internal.table.distributed.PartitionMover.movePartition(PartitionMover.java:66)
at
org.apache.ignite.internal.table.distributed.raft.RebalanceRaftGroupEventsListener.lambda$onLeaderElected$0(RebalanceRaftGroupEventsListener.java:202)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
{noformat}
> Tests from ItIgniteNodeRestartTest became flaky
> --------------------------------------------------
>
> Key: IGNITE-18737
> URL: https://issues.apache.org/jira/browse/IGNITE-18737
> Project: Ignite
> Issue Type: Bug
> Reporter: Mirza Aliev
> Assignee: Sergey Uttsel
> Priority: Major
> Labels: ignite-3
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> h3. *Motivation*
> After https://issues.apache.org/jira/browse/IGNITE-18088 test from
> ItIgniteNodeRestartTest started to fail. Need to investigate the reason of
> fails
> {noformat}
> Caused by: java.util.concurrent.CancellationException
> at
> java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396)
> at
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:491)
> at
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:473)
> at
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.refreshAndGetLeaderWithTerm(RaftGroupServiceImpl.java:232)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.ensureReplicaIsPrimary(PartitionReplicaListener.java:1880)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.invoke(PartitionReplicaListener.java:275)
> at
> org.apache.ignite.internal.replicator.Replica.processRequest(Replica.java:61)
> at
> org.apache.ignite.internal.replicator.ReplicaManager.lambda$new$5(ReplicaManager.java:162)
> at
> org.apache.ignite.network.DefaultMessagingService.sendToSelf(DefaultMessagingService.java:279)
> at
> org.apache.ignite.network.DefaultMessagingService.invoke0(DefaultMessagingService.java:226)
> at
> org.apache.ignite.network.DefaultMessagingService.invoke(DefaultMessagingService.java:154)
> at
> org.apache.ignite.internal.replicator.ReplicaService.sendToReplica(ReplicaService.java:91)
> at
> org.apache.ignite.internal.replicator.ReplicaService.invoke(ReplicaService.java:178)
> at
> org.apache.ignite.internal.tx.impl.TxManagerImpl.cleanup(TxManagerImpl.java:150)
> at
> org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processTxFinishAction$42(PartitionReplicaListener.java:984)
> at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
> at
> org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$38(RaftGroupServiceImpl.java:526)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:479)
> at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
> at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
> at
> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
> at
> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
> at
> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183){noformat}
> The root cause is that distribution zones data nodes watch listener in
> TableManager invokes updatePendingAssignmentsKeys with the same data nodes as
> in table configuration assignments so RaftGroupServiceImpl stops and new the
> same RaftGroupServiceImpl starts. It cancels invokes to old
> RaftGroupServiceImpl so we have CancellationException at sendWithRetry.
> h3. *Implementation Notes*
> It is not need to change the assignments if the calculated assignments equals
> to the assignments in the table configuration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
