[
https://issues.apache.org/jira/browse/HDDS-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-11128:
------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Intermittent timeout during waitForReplicaCount in
> TestReconAndAdminContainerCLI
> --------------------------------------------------------------------------------
>
> Key: HDDS-11128
> URL: https://issues.apache.org/jira/browse/HDDS-11128
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Attila Doroszlai
> Assignee: Siyao Meng
> Priority: Minor
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> {code}
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 102.149 s <<<
> FAILURE! - in org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.testNodesInDecommissionOrMaintenance(HddsProtos$NodeOperationalState,HddsProtos$NodeOperationalState,
> boolean) Time elapsed: 36.908 s <<< ERROR!
> TimeoutException:
> ...
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:204)
> at
> org.apache.hadoop.ozone.container.TestHelper.waitForReplicaCount(TestHelper.java:444)
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.testNodesInDecommissionOrMaintenance(TestReconAndAdminContainerCLI.java:264)
> {code}
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2024/05/31/31535/it-recon/hadoop-ozone/integration-test/org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.txt
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2024/07/01/32031/it-recon/hadoop-ozone/integration-test/org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.txt
> h3. Root cause
> {{testNodesInDecommissionOrMaintenance}} times out at
> {{OzoneTestHelper.waitForReplicaCount}} after a decommission/maintenance
> step. The wait is a bare replica-count equality ({{countReplicas(...) ==
> N}}). During decommission/maintenance SCM is actively adding and
> re-evaluating replicas, so the count passes through the expected value
> transiently, and a fixed-value poll can sample the wrong instant and time
> out. HDDS-10582 previously only lowered the poll interval (1000ms -> 200ms)
> to narrow that window, so the flake recurred.
> h3. Fix
> Add {{OzoneTestHelper.waitForStableReplicaCount}}, which returns only once
> replication has quiesced ({{ReplicationManager}} reports no pending
> add/delete ops for the container) *and* the count equals N, so the assertion
> is on a settled state rather than a transient one. It uses the same 30s
> budget as {{waitForReplicaCount}}; no larger timeout is needed, because the
> call runs right after the {{DECOMMISSIONED}}/{{IN_MAINTENANCE}} gate, which
> already requires the new replica to exist, so the settle returns almost
> immediately. The existing {{waitForReplicaCount}} and its other callers are
> left untouched. The test stays {{@Flaky("HDDS-11128")}} since no wait can
> prove non-flakiness.
> * https://github.com/apache/ozone/actions/runs/31658391765/job/94319911084
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
