[
https://issues.apache.org/jira/browse/HDDS-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-16398:
----------------------------------
Status: Patch Available (was: Open)
> TestCommitWatcher#testReleaseBuffersOnException intermittently fails with
> AlreadyClosedException (NotLeaderException on freshly-allocated pipeline)
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16398
> URL: https://issues.apache.org/jira/browse/HDDS-16398
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
>
> {{org.apache.hadoop.hdds.scm.storage.TestCommitWatcher#testReleaseBuffersOnException}}
> fails intermittently in CI. The failure occurs during test setup — while
> waiting on the _first_ {{putBlock}} response ({{{}future1.get(){}}} at
> {{{}TestCommitWatcher.java:262{}}}), before the test reaches the datanode
> shutdowns that exercise the behavior actually under test.
> *Stack trace*
> {code} java.util.concurrent.ExecutionException:
> org.apache.ratis.protocol.exceptions.AlreadyClosedException:
> SlidingWindow$Client:client-B2BDCC4ECA19->RAFT is closed. at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2094)
> at
> org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.testReleaseBuffersOnException(TestCommitWatcher.java:262)
> Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException:
> SlidingWindow$Client:client-... is closed. at
> org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:409)
> ... Caused by:
> org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed
> RaftClientRequest: client-...->d62e25aa-...@group-..., cid=29, seq=1*,
> Watch(0), null for 3 attempts with RequestTypeDependentRetryPolicy\{ ... } at
> org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:352)
> Caused by: org.apache.ratis.protocol.exceptions.NotLeaderException: Server
> d62e25aa-... is not the leader, suggested leader is:
> 59f5747a-...|127.0.0.1:15071 at
> org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:402)
> \{code}
> *Root cause*
> The test allocates a fresh RATIS THREE pipeline and immediately issues async
> write/putBlock commands:
> {code} ContainerWithPipeline container = storageContainerLocationClient
> .allocateContainer(RATIS, THREE, OZONE); Pipeline pipeline =
> container.getPipeline(); ... ratisClient.sendCommandAsync(writeChunkRequest);
> XceiverClientReply reply = ratisClient.sendCommandAsync(putBlockRequest);
> \{code}
> {{cluster.waitForClusterToBeReady()}} (called in {{{}@BeforeEach{}}}) waits
> for datanode registration and SCM readiness, but does *not* guarantee that
> the newly-allocated Ratis group has completed leader election. When the write
> lands on a node that is not yet the leader, Ratis returns
> {{{}NotLeaderException{}}}. Under the aggressive per-test retry policy (short
> watch timeout, limited attempts), the request exhausts its retries and the
> client is closed, surfacing as {{AlreadyClosedException}} from
> {{{}future1.get(){}}}.
> This is a timing/leader-election race in test setup, not a product defect in
> the commit-watch path.
> *Reproducibility*
> Intermittent. Timing-dependent; does not reproduce reliably. This test has a
> long history of the same class of flakiness — see related issues below
> (HDDS-9766 reports the identical {{AlreadyClosedException}} signature).
> *Suggested fix*
> Wait for the allocated pipeline to have an elected leader before issuing
> writes — e.g., block on pipeline readiness / leader election
> ({{{}XceiverClientRatis.watchForCommit{}}} on the leader, or a
> {{{}waitForPipelineReady{}}}-style helper) after {{allocateContainer}} and
> before the first {{{}sendCommandAsync{}}}, so setup no longer races election.
> Apply the same guard to {{{}testReleaseBuffers{}}}, which shares the pattern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
