Siyao Meng created HDDS-17311:
---------------------------------
Summary: Intermittent failure in TestDeleteContainerHandler when
the first write races the Ratis leader becoming ready
Key: HDDS-17311
URL: https://issues.apache.org/jira/browse/HDDS-17311
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
Tests run: 6, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 254.5 s <<<
FAILURE! -- in
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteContainerRequestHandlerOnClosedContainer
-- Time elapsed: 65.08 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition.
...
at
org.apache.hadoop.ozone.OzoneTestUtils.waitBlockDeleted(OzoneTestUtils.java:186)
at
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteContainerRequestHandlerOnClosedContainer(TestDeleteContainerHandler.java:649)
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteNonEmptyContainerOnDirEmptyCheckTrue
-- Time elapsed: 63.08 s <<< ERROR!
at
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteNonEmptyContainerOnDirEmptyCheckTrue(TestDeleteContainerHandler.java:196)
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteNonEmptyContainerOnDirEmptyCheckFalse
-- Time elapsed: 63.06 s <<< ERROR!
at
org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteNonEmptyContainerOnDirEmptyCheckFalse(TestDeleteContainerHandler.java:314)
{code}
h3. Root cause
Same race as HDDS-17213. {{setup}} only waits for SCM to report an OPEN
RATIS/ONE pipeline ({{waitForPipelineTobeReady}}), and the first key write of
the class reached the new Ratis group before its leader was ready:
* WriteChunk (seq=2) and PutBlock (seq=3) for container #1 were both rejected
with {{LeaderNotReadyException}}. On retry the PutBlock was appended and
applied without its WriteChunk, failed with "Chunk file not found", and the
replica of container #1 was marked UNHEALTHY. The client then wrote the key to
container #2 on a new pipeline.
* On key commit, OM sent the uncommitted block of container #1 to SCM for
deletion. Container #1 stays CLOSING in SCM with an UNHEALTHY replica, so
{{DeletedBlockLogImpl#getTransactions}} skips that transaction on every run
({{checkInadequateReplica}}) and it is never removed.
* All tests share one cluster ({{@BeforeAll}}). With that stale transaction in
the log, {{OzoneTestUtils#flushAndWaitForDeletedBlockLog}} returns before the
test's own transaction is flushed, and {{OzoneTestUtils#waitBlockDeleted}}
(waits for zero transactions) times out after 60 s in every later test that
calls it.
h3. Fix
In {{setup}}, after the pipeline is OPEN, wait until the datanode's Ratis
division of that pipeline reports {{isLeaderReady()}} before the first write,
as done for {{TestFinalizeBlock}} in HDDS-17213.
- https://github.com/apache/ozone/actions/runs/39747650064/job/119730210068
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
