[
https://issues.apache.org/jira/browse/HDDS-17346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17346:
------------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
> Intermittent timeout in container scanner integration tests waiting for
> container close
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-17346
> URL: https://issues.apache.org/jira/browse/HDDS-17346
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> [ERROR]
> org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(ContainerTestCorruptions)[2]
> -- Time elapsed: 5.119 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition. Thread diagnostics:
> ...
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.closeContainerAndWait(ContainerScannerIntegrationTests.java:169)
> at
> org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:162)
> at
> org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:151)
> at
> org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(TestBackgroundContainerMetadataScannerIntegration.java:96)
> {code}
> h3. Root cause
> {{ContainerScannerIntegrationTests#closeContainerAndWait}} asks SCM to close
> the container, then gives the datanode only 5 seconds to report it CLOSED.
> The close is applied through Ratis, and {{KeyValueContainer#close}} calls
> {{flushAndSyncDB}}, which fsyncs the RocksDB WAL. In the failed run SCM
> triggered the close of container #3 at 11:32:56.5. At the timeout
> (11:33:01.5) the thread dump shows the {{ContainerOp}} thread still in
> {{RocksDB.flushWal}} under {{KeyValueContainer.close}}. The container was
> closed at 11:33:03.0, about 6.5 seconds after the close request. The first
> case's close in the same run already took about 3 seconds, so 5 seconds is
> not enough headroom on a slow CI disk.
> h3. Fix
> Raise the timeout of the datanode close wait in {{closeContainerAndWait}}
> from 5 to 30 seconds. The wait polls, so passing runs finish just as fast.
> The helper is shared by all container scanner integration tests.
> - https://github.com/apache/ozone/actions/runs/39763337594/job/119785408645
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
