Siyao Meng created HDDS-16934:
---------------------------------
Summary: Intermittent failure in
TestSCMNodeManager#testDatanodeFinalizedCounterExcludesNonHealthyNodes
Key: HDDS-16934
URL: https://issues.apache.org/jira/browse/HDDS-16934
Project: Apache Ozone
Issue Type: Sub-task
Components: SCM, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
[ERROR]
org.apache.hadoop.hdds.scm.node.TestSCMNodeManager.testDatanodeFinalizedCounterExcludesNonHealthyNodes(NodeStatus)[2]
<<< FAILURE!
org.opentest4j.AssertionFailedError: IN_SERVICE(no expiry)-DEAD node should be
excluded from total count ==> expected: <1> but was: <0>
{code}
Observed locally, about once in seven runs, only for the DEAD parameter.
h3. Root cause
The test (added in HDDS-15034) sets the stale node interval to 3s and the dead
node interval to 6s, and drives the node state with fixed {{Thread.sleep}}
calls. In the DEAD case the last heartbeat of the baseline node is followed by
{{Thread.sleep(3000)}}, so that heartbeat is at least 3s old when the
assertions run. {{NodeStateManager#checkNodesHealth}} only treats a node as
healthy when its last heartbeat is within the stale node interval. If the
health check (100ms period) fires between the end of the sleep and the
assertion, the baseline node itself is marked STALE and
{{getTotalHealthyDatanodes()}} returns 0 instead of 1.
Adding a 150ms sleep before the assertions makes the DEAD case fail every time,
with the baseline node reported as STALE.
h3. Fix
Replace the fixed sleeps with {{GenericTestUtils.waitFor}} that keeps
heartbeating the baseline node until the other node reaches the expected state,
as other tests in the same class already do.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
