[
https://issues.apache.org/jira/browse/HDDS-16235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai resolved HDDS-16235.
-------------------------------------
Resolution: Duplicate
> Intermittent failure in
> TestOzoneClientRetriesOnExceptions#testMaxRetriesByOzoneClient
> --------------------------------------------------------------------------------------
>
> Key: HDDS-16235
> URL: https://issues.apache.org/jira/browse/HDDS-16235
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Meng-Shuan Tsai
> Assignee: Meng-Shuan Tsai
> Priority: Major
>
> testMaxRetriesByOzoneClient fails intermittently (~3% of runs) at
> TestOzoneClientRetriesOnExceptions.java:206 with:
> {code:java}
> java.io.IOException: Failed to execute command cmdType: CreateContainer
> containerID: 1 datanodeUuid: "..." createContainer { state: OPEN }
> at
> org.apache.hadoop.hdds.scm.XceiverClientSpi.getIOExceptionForSendCommand(XceiverClientSpi.java:178)
> at
> org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:131)
> at
> org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnExceptions.testMaxRetriesByOzoneClient(TestOzoneClientRetriesOnExceptions.java:206)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.ratis.protocol.exceptions.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Container creation failed because key value container already exists
> at
> org.apache.hadoop.hdds.scm.XceiverClientSpi.sendCommand(XceiverClientSpi.java:125)
> Caused by: org.apache.ratis.protocol.exceptions.StateMachineException:
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> Container creation failed because key value container already exists
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$applyTransaction$2(ContainerStateMachine.java:1193)
> {code}
> Steps to reproduce: run TestOzoneClientRetriesOnExceptions out of 100 runs (
> [https://github.com/shuan1026/ozone/actions/runs/32260061359]), 3 failed with
> this identical stack trace, all on containerID 1. Datanode-side log for one
> such failure:
> {code:java}
> 14:18:42,671 leader elected for the container's pipeline (term 1)
> 14:18:42,788 CreateContainer for containerID 1 fails: Result:
> CONTAINER_ALREADY_EXISTS
> 14:18:42,800 ContainerStateMachine ApplyTransaction failed at logIndex 5
> 14:18:42,807 pipeline close action triggered as a result
> {code}
> Line 206 is the test's own manual
> xceiverClient.sendCommand(getCreateContainerRequest(...)), sent once per
> pre-allocated container to ensure it is OPEN before writing. By design this
> should be the only path that creates the container on the datanode, but the
> log shows it was already created by something else ~117ms after the
> pipeline's leader election, so the test's own call is rejected as a
> duplicate, non-idempotent create.
> No @Flaky annotation exists on this test and no prior JIRA report was found
> for it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
