[
https://issues.apache.org/jira/browse/HDDS-17311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chi-Hsuan Huang updated HDDS-17311:
-----------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Intermittent failure in TestDeleteContainerHandler when the first write races
> the Ratis leader becoming ready
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17311
> URL: https://issues.apache.org/jira/browse/HDDS-17311
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> {code}
> Tests run: 6, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 254.5 s <<<
> FAILURE! -- in
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteContainerRequestHandlerOnClosedContainer
> -- Time elapsed: 65.08 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition.
> ...
> at
> org.apache.hadoop.ozone.OzoneTestUtils.waitBlockDeleted(OzoneTestUtils.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteContainerRequestHandlerOnClosedContainer(TestDeleteContainerHandler.java:649)
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteNonEmptyContainerOnDirEmptyCheckTrue
> -- Time elapsed: 63.08 s <<< ERROR!
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteNonEmptyContainerOnDirEmptyCheckTrue(TestDeleteContainerHandler.java:196)
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteNonEmptyContainerOnDirEmptyCheckFalse
> -- Time elapsed: 63.06 s <<< ERROR!
> at
> org.apache.hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler.testDeleteNonEmptyContainerOnDirEmptyCheckFalse(TestDeleteContainerHandler.java:314)
> {code}
> h3. Root cause
> Same race as HDDS-17213. {{setup}} only waits for SCM to report an OPEN
> RATIS/ONE pipeline ({{waitForPipelineTobeReady}}), and the first key write of
> the class reached the new Ratis group before its leader was ready:
> * WriteChunk (seq=2) and PutBlock (seq=3) for container #1 were both rejected
> with {{LeaderNotReadyException}}. On retry the PutBlock was appended and
> applied without its WriteChunk, failed with "Chunk file not found", and the
> replica of container #1 was marked UNHEALTHY. The client then wrote the key
> to container #2 on a new pipeline.
> * On key commit, OM sent the uncommitted block of container #1 to SCM for
> deletion. Container #1 stays CLOSING in SCM with an UNHEALTHY replica, so
> {{DeletedBlockLogImpl#getTransactions}} skips that transaction on every run
> ({{checkInadequateReplica}}) and it is never removed.
> * All tests share one cluster ({{@BeforeAll}}). With that stale transaction
> in the log, {{OzoneTestUtils#flushAndWaitForDeletedBlockLog}} returns before
> the test's own transaction is flushed, and
> {{OzoneTestUtils#waitBlockDeleted}} (waits for zero transactions) times out
> after 60 s in every later test that calls it.
> h3. Fix
> In {{setup}}, after the pipeline is OPEN, wait until the datanode's Ratis
> division of that pipeline reports {{isLeaderReady()}} before the first write,
> as done for {{TestFinalizeBlock}} in HDDS-17213.
> - https://github.com/apache/ozone/actions/runs/39747650064/job/119730210068
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
