[
https://issues.apache.org/jira/browse/HDDS-16967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128021#comment-18128021
]
Siyao Meng commented on HDDS-16967:
-----------------------------------
Second failure mode, seen twice on master:
{code}
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException
<<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
at
org.apache.hadoop.ozone.container.OzoneTestHelper.waitForContainerClose(OzoneTestHelper.java:363)
{code}
{{OzoneTestHelper#waitForContainerClose}} fired {{CLOSE_CONTAINER}} inside the
per-datanode loop, right after checking the first replica. A restarted follower
that was still catching up created the container and applied the close
together, so the check for that replica found it already closed. The PR now
fires the close once per container, after every replica is present and open.
- https://github.com/apache/ozone/actions/runs/38924347415/job/117067399755
- https://github.com/apache/ozone/actions/runs/38964992436/job/117194652841
> Intermittent failure in
> TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16967
> URL: https://issues.apache.org/jira/browse/HDDS-16967
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Client, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 295.5 s <<<
> FAILURE! -- in org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient
> org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException
> -- Time elapsed: 127.8 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition.
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> org.apache.hadoop.ozone.container.OzoneTestHelper.waitForContainerClose(OzoneTestHelper.java:358)
> at
> org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException(TestFailureHandlingByClient.java:412)
> {code}
> h3. Root cause
> The previous test in the class shut down two datanodes of a pipeline, and
> {{restartDownDataNodes}} restarted them before this test.
> {{waitForClusterToBeReady}} only waits for the datanodes to register with
> SCM; it does not wait for their Ratis groups to elect a leader. In the
> failing run the surviving peer of that pipeline kept getting {{UNAVAILABLE:
> io exception}} on vote requests to the restarted peers, so the group had no
> leader from 03:41:58 until 03:42:36.
> SCM still considered the pipeline open, so the first block of the test key
> (container 4) was allocated on it. The write timed out with
> {{RaftRetryFailureException}} after about 27s and the client moved the data
> to a new block on another pipeline. The test then read the container ID from
> {{streamEntryList.get(0)}}, which is still container 4. That container was
> never created on any datanode, so {{waitForContainerClose}} timed out waiting
> for it to be present (100s).
> The same leaderless pipeline can also break the other tests in the class,
> which assume their first block lands on a healthy pipeline.
> h3. Fix
> In {{restartDownDataNodes}}, after the cluster is ready, wait until every
> Ratis group on each restarted datanode has a leader before the next test
> starts writing.
> - https://github.com/apache/ozone/actions/runs/38341370639/job/115115959763
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
