Kirill Tkalenko created IGNITE-24662:
----------------------------------------
Summary: Fail of one of the node can lead to errors: it is
impossible to subscribe to the replica
Key: IGNITE-24662
URL: https://issues.apache.org/jira/browse/IGNITE-24662
Project: Ignite
Issue Type: Bug
Reporter: Kirill Tkalenko
When testing the cluster node restart scenario, I found the errors below in the
logs; we need to sort them out.
{noformat}
2025-02-27 11:09:27:433 +0000
[ERROR][%poc-tester-SERVER-192.168.208.141-id-0%Raft-Group-Client-9][TopologyAwareRaftGroupService]
Could not send the subscribe message to the node
[node={id=4eeb85bd-ac67-4507-b503-70634af9cf11,
name=poc-tester-SERVER-192.168.211.11-id-0, address=192.168.211.11:3344},
msg=SubscriptionLeaderChangeRequestImpl [groupId=22_part_1, subscribe=true]]
java.util.concurrent.CompletionException:
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.ignite.internal.network.OutNetworkObject.failAcknowledgement(OutNetworkObject.java:95)
at
org.apache.ignite.internal.network.recovery.RecoveryDescriptor.dispose(RecoveryDescriptor.java:265)
at
org.apache.ignite.internal.network.netty.ConnectionManager.blockAndDisposeDescriptor(ConnectionManager.java:700)
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:668)
at
org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:644)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.ignite.internal.network.RecipientLeftException:
IGN-NETWORK-5 TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:665)
... 7 more
2025-02-27 11:09:27:436 +0000
[ERROR][%poc-tester-SERVER-192.168.208.141-id-0%Raft-Group-Client-9][ReplicaManager]
Rebalance failover subscription on elected primary replica failed
[groupId=22_part_1].
java.util.concurrent.CompletionException:
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.ignite.internal.network.OutNetworkObject.failAcknowledgement(OutNetworkObject.java:95)
at
org.apache.ignite.internal.network.recovery.RecoveryDescriptor.dispose(RecoveryDescriptor.java:265)
at
org.apache.ignite.internal.network.netty.ConnectionManager.blockAndDisposeDescriptor(ConnectionManager.java:700)
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:668)
at
org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:644)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Suppressed: org.apache.ignite.lang.IgniteException: IGN-CMN-65535
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
at
org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService.lambda$subscribeLeader$3(TopologyAwareRaftGroupService.java:338)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
... 3 more
Caused by: [CIRCULAR REFERENCE:
java.util.concurrent.CompletionException:
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b]
Caused by: org.apache.ignite.internal.network.RecipientLeftException:
IGN-NETWORK-5 TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:665)
... 7 more
2025-02-27 11:09:27:437 +0000
[ERROR][%poc-tester-SERVER-192.168.208.141-id-0%Raft-Group-Client-9][FailureManager]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=CRITICAL_ERROR]
java.util.concurrent.CompletionException:
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.ignite.internal.network.OutNetworkObject.failAcknowledgement(OutNetworkObject.java:95)
at
org.apache.ignite.internal.network.recovery.RecoveryDescriptor.dispose(RecoveryDescriptor.java:265)
at
org.apache.ignite.internal.network.netty.ConnectionManager.blockAndDisposeDescriptor(ConnectionManager.java:700)
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:668)
at
org.apache.ignite.internal.network.netty.ConnectionManager.lambda$handleNodeLeft$9(ConnectionManager.java:644)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Suppressed: org.apache.ignite.lang.IgniteException: IGN-CMN-65535
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
at
org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService.lambda$subscribeLeader$3(TopologyAwareRaftGroupService.java:338)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
... 3 more
Caused by: [CIRCULAR REFERENCE:
java.util.concurrent.CompletionException:
org.apache.ignite.internal.network.RecipientLeftException: IGN-NETWORK-5
TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b]
Caused by: org.apache.ignite.internal.network.RecipientLeftException:
IGN-NETWORK-5 TraceId:aa363e39-2e72-4753-b3ca-57cbc10cff1b
at
org.apache.ignite.internal.network.netty.ConnectionManager.disposeRecoveryDescriptorsOfLeftNode(ConnectionManager.java:665)
... 7 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
