Siyao Meng created HDDS-16132:
---------------------------------
Summary: Intermittent timeout in
TestSCMFollowerCatchupWithContainerReport (follower stuck in safe mode on idle
cluster)
Key: HDDS-16132
URL: https://issues.apache.org/jira/browse/HDDS-16132
Project: Apache Ozone
Issue Type: Sub-task
Components: SCM HA, test
Reporter: Siyao Meng
{code}
[ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 311.2 s
<<< FAILURE! -- in
org.apache.hadoop.hdds.scm.TestSCMFollowerCatchupWithContainerReport
[ERROR]
org.apache.hadoop.hdds.scm.TestSCMFollowerCatchupWithContainerReport.testFollowerCatchupOnIdleCluster
-- Time elapsed: 125.6 s <<< ERROR!
java.util.concurrent.TimeoutException:
Timed out waiting for condition.
...
[ERROR]
org.apache.hadoop.hdds.scm.TestSCMFollowerCatchupWithContainerReport.testFollowerCatchupAfterContainerCreate
-- Time elapsed: 121.0 s <<< ERROR!
java.util.concurrent.TimeoutException:
{code}
Both failing methods time out at {{waitFor(() -> !newFollower.isInSafeMode(),
...)}} after restarting the follower SCM. In the logs the restarted follower
stays in safe mode for the full 120s with {{registered datanodes (=0) >=
required datanodes (=3)}}: no datanode ever registers because the follower's
{{SCMDatanodeProtocolServer}} is never started.
Possible cause (needs confirmation from the HDDS-14989 author): HDDS-14989
defers starting the follower's datanode RPC server to
{{SCMStateMachine.tryStartDNServerAndRefreshSafeMode()}}, which is invoked only
from Ratis event callbacks ({{applyTransaction}}, {{notifyLeaderChanged}},
{{notifyTermIndexUpdated}}) and is gated on {{isFollowerCaughtUp()}}. On a
restarted follower over an idle cluster, {{notifyLeaderChanged}} can fire once
while {{getLeaderCommitIndex()}} still returns -1, so the DN server is not
started; with no further transactions there is no later callback to retry, and
safe mode never clears. This looks like a product-side gap in the idle-follower
path rather than pure CI flakiness (raising the 120s wait does not help).
Seen in:
-
https://github.com/adoroszlai/ozone-build-results/blob/master/2026/08/08/52075/integration-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.TestSCMFollowerCatchupWithContainerReport.txt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
