[
https://issues.apache.org/jira/browse/HDDS-16781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16781:
------------------------------
Summary: Intermittent ReadChunk count mismatch in TestKeyInputStream (was:
Intermittent failure in TestKeyInputStream#testSkip)
> Intermittent ReadChunk count mismatch in TestKeyInputStream
> -----------------------------------------------------------
>
> Key: HDDS-16781
> URL: https://issues.apache.org/jira/browse/HDDS-16781
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Client, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR]
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(ContainerLayoutVersion)[2]
> -- Time elapsed: 22.70 s <<< FAILURE!
> org.opentest4j.AssertionFailedError: expected: <2> but was: <4>
> at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:632)
> at
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testSkip(TestKeyInputStream.java:368)
> at
> org.apache.hadoop.ozone.client.rpc.read.TestKeyInputStream.testNonReplicationReads(TestKeyInputStream.java:136)
> {code}
> h3. Root cause
> {{testSkip}} writes a RATIS/THREE key and immediately reads it, asserting
> exactly 2 ReadChunk ops via {{XceiverClientMetrics}}. The write returns once
> the data is committed, but a follower may not have applied the PutBlock yet.
> In the failing run both ReadChunk requests first went to such a follower:
> {code}
> WARN keyvalue.KeyValueHandler - Operation: ReadChunk , Message: Unable to
> find the block with bcsID 485. Container 9 bcsId is 483. , Result:
> UNKNOWN_BCSID
> {code}
> The client correctly retried on another datanode, but the metric counts every
> attempt, so it saw 4 ReadChunk ops (2 failed and 2 successful).
> h3. Fix
> Close the key's container and wait until it is CLOSED on all replicas before
> reading. Followers apply the close only after the PutBlock (Ratis log order),
> so no replica can lag behind the block's bcsId. {{testSeek}} has the same
> exact ReadChunk count assertion after a write, so the wait is applied there
> too.
> - https://github.com/apache/ozone/actions/runs/37727947048/job/113153515921
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
