[
https://issues.apache.org/jira/browse/HDDS-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-11128:
------------------------------
Description:
{code}
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 102.149 s <<<
FAILURE! - in org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.testNodesInDecommissionOrMaintenance(HddsProtos$NodeOperationalState,HddsProtos$NodeOperationalState,
boolean) Time elapsed: 36.908 s <<< ERROR!
TimeoutException:
...
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:204)
at
org.apache.hadoop.ozone.container.TestHelper.waitForReplicaCount(TestHelper.java:444)
at
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.testNodesInDecommissionOrMaintenance(TestReconAndAdminContainerCLI.java:264)
{code}
*
https://github.com/adoroszlai/ozone-build-results/blob/master/2024/05/31/31535/it-recon/hadoop-ozone/integration-test/org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.txt
*
https://github.com/adoroszlai/ozone-build-results/blob/master/2024/07/01/32031/it-recon/hadoop-ozone/integration-test/org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.txt
h3. Root cause
{{testNodesInDecommissionOrMaintenance}} times out at
{{OzoneTestHelper.waitForReplicaCount}} after a decommission/maintenance step.
The wait is a bare replica-count equality ({{countReplicas(...) == N}}). During
decommission/maintenance SCM is actively adding and re-evaluating replicas, so
the count passes through the expected value transiently, and a fixed-value poll
can sample the wrong instant and time out. HDDS-10582 previously only lowered
the poll interval (1000ms -> 200ms) to narrow that window, so the flake
recurred.
h3. Fix
Add {{OzoneTestHelper.waitForStableReplicaCount}}, which returns only once
replication has quiesced ({{ReplicationManager}} reports no pending add/delete
ops for the container) *and* the count equals N, so the assertion is on a
settled state rather than a transient one. It uses the same 30s budget as
{{waitForReplicaCount}}; no larger timeout is needed, because the call runs
right after the {{DECOMMISSIONED}}/{{IN_MAINTENANCE}} gate, which already
requires the new replica to exist, so the settle returns almost immediately.
The existing {{waitForReplicaCount}} and its other callers are left untouched.
The test stays {{@Flaky("HDDS-11128")}} since no wait can prove non-flakiness.
* https://github.com/apache/ozone/actions/runs/31658391765/job/94319911084
was:
{code}
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 102.149 s <<<
FAILURE! - in org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.testNodesInDecommissionOrMaintenance(HddsProtos$NodeOperationalState,HddsProtos$NodeOperationalState,
boolean) Time elapsed: 36.908 s <<< ERROR!
TimeoutException:
...
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:204)
at
org.apache.hadoop.ozone.container.TestHelper.waitForReplicaCount(TestHelper.java:444)
at
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.testNodesInDecommissionOrMaintenance(TestReconAndAdminContainerCLI.java:264)
{code}
*
https://github.com/adoroszlai/ozone-build-results/blob/master/2024/05/31/31535/it-recon/hadoop-ozone/integration-test/org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.txt
*
https://github.com/adoroszlai/ozone-build-results/blob/master/2024/07/01/32031/it-recon/hadoop-ozone/integration-test/org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.txt
h3. Root cause
{{testNodesInDecommissionOrMaintenance}} (decommission case) times out at
{{OzoneTestHelper.waitForReplicaCount}} waiting for a decommission-triggered
replica copy (3 -> 4 for the first node, 4 -> 5 for the second) to be reflected
in SCM. The shared {{waitForReplicaCount}} helper used a fixed 30s budget for
all callers. HDDS-10582 only lowered its poll interval (1000ms -> 200ms) and
kept the 30s total, so on a loaded CI runner the replica copy is not always
observed in time and the test flakes.
h3. Fix
Add a 4-arg {{waitForReplicaCount(containerID, count, cluster, timeoutMillis)}}
overload; the existing 3-arg method delegates with the same 30s default (no
behavior change for the other callers). The decommission/maintenance
replica-copy waits in {{TestReconAndAdminContainerCLI}} now use a 60s budget
(double the previous 30s; the happy path returns as soon as the copy lands, so
a tighter budget keeps the failing path bounded since this test is rerun on
failure in the flaky split). The test stays {{@Flaky("HDDS-11128")}} since a
larger timeout cannot prove non-flakiness.
* https://github.com/apache/ozone/actions/runs/31658391765/job/94319911084
> Intermittent timeout during waitForReplicaCount in
> TestReconAndAdminContainerCLI
> --------------------------------------------------------------------------------
>
> Key: HDDS-11128
> URL: https://issues.apache.org/jira/browse/HDDS-11128
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test
> Reporter: Attila Doroszlai
> Assignee: Siyao Meng
> Priority: Minor
> Labels: pull-request-available
>
> {code}
> Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 102.149 s <<<
> FAILURE! - in org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.testNodesInDecommissionOrMaintenance(HddsProtos$NodeOperationalState,HddsProtos$NodeOperationalState,
> boolean) Time elapsed: 36.908 s <<< ERROR!
> TimeoutException:
> ...
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:204)
> at
> org.apache.hadoop.ozone.container.TestHelper.waitForReplicaCount(TestHelper.java:444)
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.testNodesInDecommissionOrMaintenance(TestReconAndAdminContainerCLI.java:264)
> {code}
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2024/05/31/31535/it-recon/hadoop-ozone/integration-test/org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.txt
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2024/07/01/32031/it-recon/hadoop-ozone/integration-test/org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.txt
> h3. Root cause
> {{testNodesInDecommissionOrMaintenance}} times out at
> {{OzoneTestHelper.waitForReplicaCount}} after a decommission/maintenance
> step. The wait is a bare replica-count equality ({{countReplicas(...) ==
> N}}). During decommission/maintenance SCM is actively adding and
> re-evaluating replicas, so the count passes through the expected value
> transiently, and a fixed-value poll can sample the wrong instant and time
> out. HDDS-10582 previously only lowered the poll interval (1000ms -> 200ms)
> to narrow that window, so the flake recurred.
> h3. Fix
> Add {{OzoneTestHelper.waitForStableReplicaCount}}, which returns only once
> replication has quiesced ({{ReplicationManager}} reports no pending
> add/delete ops for the container) *and* the count equals N, so the assertion
> is on a settled state rather than a transient one. It uses the same 30s
> budget as {{waitForReplicaCount}}; no larger timeout is needed, because the
> call runs right after the {{DECOMMISSIONED}}/{{IN_MAINTENANCE}} gate, which
> already requires the new replica to exist, so the settle returns almost
> immediately. The existing {{waitForReplicaCount}} and its other callers are
> left untouched. The test stays {{@Flaky("HDDS-11128")}} since no wait can
> prove non-flakiness.
> * https://github.com/apache/ozone/actions/runs/31658391765/job/94319911084
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
