Siyao Meng created HDDS-16967:
---------------------------------
Summary: Intermittent failure in
TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException
Key: HDDS-16967
URL: https://issues.apache.org/jira/browse/HDDS-16967
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Client, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 295.5 s <<<
FAILURE! -- in org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException
-- Time elapsed: 127.8 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition.
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
at
org.apache.hadoop.ozone.container.OzoneTestHelper.waitForContainerClose(OzoneTestHelper.java:358)
at
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException(TestFailureHandlingByClient.java:412)
{code}
h3. Root cause
The previous test in the class shut down two datanodes of a pipeline, and
{{restartDownDataNodes}} restarted them before this test.
{{waitForClusterToBeReady}} only waits for the datanodes to register with SCM;
it does not wait for their Ratis groups to elect a leader. In the failing run
the surviving peer of that pipeline kept getting {{UNAVAILABLE: io exception}}
on vote requests to the restarted peers, so the group had no leader from
03:41:58 until 03:42:36.
SCM still considered the pipeline open, so the first block of the test key
(container 4) was allocated on it. The write timed out with
{{RaftRetryFailureException}} after about 27s and the client moved the data to
a new block on another pipeline. The test then read the container ID from
{{streamEntryList.get(0)}}, which is still container 4. That container was
never created on any datanode, so {{waitForContainerClose}} timed out waiting
for it to be present (100s).
The same leaderless pipeline can also break the other tests in the class, which
assume their first block lands on a healthy pipeline.
h3. Fix
In {{restartDownDataNodes}}, after the cluster is ready, wait until every Ratis
group on each restarted datanode has a leader before the next test starts
writing.
- https://github.com/apache/ozone/actions/runs/38341370639/job/115115959763
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
