[
https://issues.apache.org/jira/browse/HDDS-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16801:
------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Intermittent failure in
> TestPerVolumePushReplication#testHealthyVolumeReplicationAfterVolumeFailure
> ---------------------------------------------------------------------------------------------------
>
> Key: HDDS-16801
> URL: https://issues.apache.org/jira/browse/HDDS-16801
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> {code}
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 101.2
> s <<< FAILURE! -- in
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication
> [ERROR]
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testHealthyVolumeReplicationAfterVolumeFailure
> -- Time elapsed: 35.31 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition. Thread diagnostics:
> ...
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.queuePushAndWaitForContainer(TestPerVolumePushReplication.java:386)
> at
> org.apache.hadoop.ozone.container.replication.TestPerVolumePushReplication.testHealthyVolumeReplicationAfterVolumeFailure(TestPerVolumePushReplication.java:219)
> {code}
> The source datanode dropped the injected push command:
> {code}
> WARN statemachine.StateContext (StateContext.java:getNextCommand(767)) -
> Detect and drop a SCMCommand replicateContainerCommand: cmdID: 1791499115402,
> encodedToken: "", term: 0, ..., containerId=1000003, ... from stale leader
> SCM, stale term 0, latest term 2.
> {code}
> h3. Root cause
> testPushAndScmReplicationWithPerVolumeEnabled restarts a datanode at the end,
> and testHealthyVolumeReplicationAfterVolumeFailure can pick that same
> datanode as the push source. queueReplicationCommand stamps the command with
> the datanode's own term via
> {{context.getTermOfLeaderSCM().ifPresent(cmd::setTerm)}}. On a just restarted
> datanode that term is not initialized yet, so the command keeps term 0. When
> StateContext later initializes the term from the max term in its command
> queue (2, from real SCM commands such as close pipeline), the term 0 push is
> dropped as coming from a stale leader SCM, and the wait for the container on
> the target times out.
> h3. Fix
> Stamp the command with the SCM leader term
> ({{getScmContext().getTermOfLeader()}}), as TestCloseContainerByPipeline and
> TestDeleteContainerHandler do, so it never depends on whether the datanode
> has learned the term yet.
> - https://github.com/apache/ozone/actions/runs/37850535429/job/113566423731
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
