Siyao Meng created HDDS-17345:
---------------------------------
Summary: Intermittent failure in
TestBackgroundContainerMetadataScannerIntegration#testCorruptionDetected
Key: HDDS-17345
URL: https://issues.apache.org/jira/browse/HDDS-17345
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 44.89 s <<<
FAILURE! -- in
org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration
org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(ContainerTestCorruptions)[2]
-- Time elapsed: 5.119 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition. Thread diagnostics:
...
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:137)
at
org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.closeContainerAndWait(ContainerScannerIntegrationTests.java:169)
at
org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:162)
at
org.apache.hadoop.ozone.dn.scanner.ContainerScannerIntegrationTests.writeDataThenCloseContainer(ContainerScannerIntegrationTests.java:151)
at
org.apache.hadoop.ozone.dn.scanner.TestBackgroundContainerMetadataScannerIntegration.testCorruptionDetected(TestBackgroundContainerMetadataScannerIntegration.java:96)
{code}
h3. Root cause
{{ContainerScannerIntegrationTests#closeContainerAndWait}} gives the datanode
only 5 seconds to report the container as CLOSED after asking SCM to close it.
That window covers SCM sending the close command (ReplicationManager runs every
second) plus the datanode applying the close through Ratis, which includes
{{KeyValueContainer#flushAndSyncDB}} (RocksDB {{flushWal}} with sync). In the
failed run the close event for container #3 was triggered at 11:32:56.5, the
thread dump at the timeout shows the ContainerOp thread still inside
{{RocksDB.flushWal}}, and the container became CLOSED at 11:33:03.0, about 6.5
seconds later. The close itself succeeded; the wait was just too short for a
slow fsync on the CI runner.
h3. Fix
Raise the close wait timeout in {{closeContainerAndWait}} from 5 to 30 seconds.
The helper is shared by all scanner integration tests.
- https://github.com/apache/ozone/actions/runs/39763337594/job/119785408645
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
