[
https://issues.apache.org/jira/browse/HDDS-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17345:
----------------------------------
Labels: pull-request-available (was: )
> Intermittent failure in
> TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-17345
> URL: https://issues.apache.org/jira/browse/HDDS-17345
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 44.89 s <<<
> FAILURE! -- in
> org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration
> org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(ContainerTestCorruptions)[2]
> -- Time elapsed: 5.119 s <<< ERROR!
> java.util.concurrent.TimeoutException:
> Timed out waiting for condition. Thread diagnostics:
> ...
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
> at
> org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.closeContainerAndWait(ContainerScannerIntegrationTests.java:169)
> at
> org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:162)
> at
> org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:151)
> at
> org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(TestBackgroundContainerMetadataScannerIntegration.java:96)
> {code}
> h3. Root cause
> {{ContainerScannerIntegrationTests#closeContainerAndWait}} gives the datanode
> only 5 seconds to report the container as CLOSED after asking SCM to close
> it. That window covers SCM sending the close command (ReplicationManager runs
> every second) plus the datanode applying the close through Ratis, which
> includes {{KeyValueContainer#flushAndSyncDB}} (RocksDB {{flushWal}} with
> sync). In the failed run the close event for container #3 was triggered at
> 11:32:56.5, the thread dump at the timeout shows the ContainerOp thread still
> inside {{RocksDB.flushWal}}, and the container became CLOSED at 11:33:03.0,
> about 6.5 seconds later. The close itself succeeded; the wait was just too
> short for a slow fsync on the CI runner.
> h3. Fix
> Raise the close wait timeout in {{closeContainerAndWait}} from 5 to 30
> seconds. The helper is shared by all scanner integration tests.
> - https://github.com/apache/ozone/actions/runs/39763337594/job/119785408645
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
