[
https://issues.apache.org/jira/browse/IGNITE-21566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-21566:
-------------------------------------
Description:
Multiple tests fail with either `Failed to get the primary replica` or becauase
corresponding lease is not accepted.
{code:java}
Caused by:
org.apache.ignite.internal.placementdriver.PrimaryReplicaAwaitTimeoutException:
IGN-PLACEMENTDRIVER-1 TraceId:11dd6a63-2686-4dfe-b1d4-ea85f3859816 The primary
replica await timed out [replicationGroupId=7_part_0,
referenceTimestamp=HybridTimestamp [physical=2024-02-19 11:58:08:158 +0000,
logical=3, composite=111958025054322691], currentLease=Lease
[leaseholder=itrst_sirot_0, leaseholderId=f43432e9-f374-41e1-b8b6-9813bb59b8a1,
accepted=false, startTime=HybridTimestamp [physical=2024-02-19 11:58:07:936
+0000, logical=2, composite=111958025039773698], expirationTime=HybridTimestamp
[physical=2024-02-19 12:00:07:936 +0000, logical=0,
composite=111958032904093696], prolongable=false,
replicationGroupId=7_part_0]]{code}
[ItTableRaftSnapshotsTest#snapshotInstallationRepeatsOnTimeout|https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7866010?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandCode+Inspection=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true]
h3. Upd#1
While starting replicas we have following code
{code:java}
if (localMemberAssignment == null || !startedRaftNode ||
replicaMgr.isReplicaStarted(replicaGrpId)) {
return;
}
try {
startReplicaWithNewListener(
... {code}
that actually won't start a replica if
{{replicaMgr.isReplicaStarted(replicaGrpId)}} that basically checks
{code:java}
public boolean isReplicaStarted(ReplicationGroupId replicaGrpId) {
return replicas.containsKey(replicaGrpId);
} {code}
where \{{replicas }}is
{code:java}
private final ConcurrentHashMap<ReplicationGroupId, CompletableFuture<Replica>>
replicas = new ConcurrentHashMap<>(); {code}
On the other hand common replica message processing pattern assumes that
replica may not be ready and in that case it populates {{replicas}} with future
and awaits it. E.g. for leaseGrantMessage
{code:java}
CompletableFuture<Replica> replicaFut = replicas.computeIfAbsent(msg.groupId(),
k -> new CompletableFuture<>());
replicaFut.thenCompose(replica -> replica.processPlacementDriverMessage(msg))
{code}
All in all that means, that if leaseGrantMessage is a bit faster than replica
start (which is fine), it will populate replicas with a future and replica
start flow won't actually start the one and complete given future but just will
incorrectly assume that replica is already started.
was:
Multiple tests fail with either `Failed to get the primary replica` or becauase
corresponding lease is not accepted.
{code:java}
Caused by:
org.apache.ignite.internal.placementdriver.PrimaryReplicaAwaitTimeoutException:
IGN-PLACEMENTDRIVER-1 TraceId:11dd6a63-2686-4dfe-b1d4-ea85f3859816 The primary
replica await timed out [replicationGroupId=7_part_0,
referenceTimestamp=HybridTimestamp [physical=2024-02-19 11:58:08:158 +0000,
logical=3, composite=111958025054322691], currentLease=Lease
[leaseholder=itrst_sirot_0, leaseholderId=f43432e9-f374-41e1-b8b6-9813bb59b8a1,
accepted=false, startTime=HybridTimestamp [physical=2024-02-19 11:58:07:936
+0000, logical=2, composite=111958025039773698], expirationTime=HybridTimestamp
[physical=2024-02-19 12:00:07:936 +0000, logical=0,
composite=111958032904093696], prolongable=false,
replicationGroupId=7_part_0]]{code}
[ItTableRaftSnapshotsTest#snapshotInstallationRepeatsOnTimeout|https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7866010?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandCode+Inspection=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true]
h3. Upd#1
While starting replicas we have following code
{code:java}
if (localMemberAssignment == null || !startedRaftNode ||
replicaMgr.isReplicaStarted(replicaGrpId)) {
return;
}
try {
startReplicaWithNewListener(
... {code}
that actually won't start a replica if
{{replicaMgr.isReplicaStarted(replicaGrpId)}} that basically checks
{code:java}
public boolean isReplicaStarted(ReplicationGroupId replicaGrpId) {
return replicas.containsKey(replicaGrpId);
} {code}
where {{replicas }}is
{code:java}
private final ConcurrentHashMap<ReplicationGroupId, CompletableFuture<Replica>>
replicas = new ConcurrentHashMap<>(); {code}
On the other hand common replica message processing pattern assumes that
replica may not be ready and in that case it populates {{replicas}} with future
and awaits it. E.g. for leaseGrantMessage
{code:java}
CompletableFuture<Replica> replicaFut = replicas.computeIfAbsent(msg.groupId(),
k -> new CompletableFuture<>());
replicaFut.thenCompose(replica -> replica.processPlacementDriverMessage(msg))
{code}
All in all that means, that if leaseGrantMessage is a bit faster than replica
start (which is fine), it will populate replicas with a future and replica
start flow won't actually start the one and complete given future but just will
incorrectly assume that replica is already started.
> Multiple tests fail with Failed to get the primary replica caused by
> PrimaryReplicaAwaitTimeoutException because of unaccepted lease
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-21566
> URL: https://issues.apache.org/jira/browse/IGNITE-21566
> Project: Ignite
> Issue Type: Bug
> Reporter: Alexander Lapin
> Assignee: Alexander Lapin
> Priority: Blocker
> Labels: ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Multiple tests fail with either `Failed to get the primary replica` or
> becauase corresponding lease is not accepted.
> {code:java}
> Caused by:
> org.apache.ignite.internal.placementdriver.PrimaryReplicaAwaitTimeoutException:
> IGN-PLACEMENTDRIVER-1 TraceId:11dd6a63-2686-4dfe-b1d4-ea85f3859816 The
> primary replica await timed out [replicationGroupId=7_part_0,
> referenceTimestamp=HybridTimestamp [physical=2024-02-19 11:58:08:158 +0000,
> logical=3, composite=111958025054322691], currentLease=Lease
> [leaseholder=itrst_sirot_0,
> leaseholderId=f43432e9-f374-41e1-b8b6-9813bb59b8a1, accepted=false,
> startTime=HybridTimestamp [physical=2024-02-19 11:58:07:936 +0000, logical=2,
> composite=111958025039773698], expirationTime=HybridTimestamp
> [physical=2024-02-19 12:00:07:936 +0000, logical=0,
> composite=111958032904093696], prolongable=false,
> replicationGroupId=7_part_0]]{code}
> [ItTableRaftSnapshotsTest#snapshotInstallationRepeatsOnTimeout|https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/7866010?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandCode+Inspection=true&expandBuildProblemsSection=true&expandBuildChangesSection=true&expandBuildTestsSection=true]
> h3. Upd#1
> While starting replicas we have following code
> {code:java}
> if (localMemberAssignment == null || !startedRaftNode ||
> replicaMgr.isReplicaStarted(replicaGrpId)) {
> return;
> }
> try {
> startReplicaWithNewListener(
> ... {code}
> that actually won't start a replica if
> {{replicaMgr.isReplicaStarted(replicaGrpId)}} that basically checks
> {code:java}
> public boolean isReplicaStarted(ReplicationGroupId replicaGrpId) {
> return replicas.containsKey(replicaGrpId);
> } {code}
> where \{{replicas }}is
> {code:java}
> private final ConcurrentHashMap<ReplicationGroupId,
> CompletableFuture<Replica>> replicas = new ConcurrentHashMap<>(); {code}
> On the other hand common replica message processing pattern assumes that
> replica may not be ready and in that case it populates {{replicas}} with
> future and awaits it. E.g. for leaseGrantMessage
>
> {code:java}
> CompletableFuture<Replica> replicaFut =
> replicas.computeIfAbsent(msg.groupId(), k -> new CompletableFuture<>());
> replicaFut.thenCompose(replica -> replica.processPlacementDriverMessage(msg))
> {code}
> All in all that means, that if leaseGrantMessage is a bit faster than replica
> start (which is fine), it will populate replicas with a future and replica
> start flow won't actually start the one and complete given future but just
> will incorrectly assume that replica is already started.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
