[
https://issues.apache.org/jira/browse/HDDS-16623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16623:
----------------------------------
Labels: pull-request-available (was: )
> Datanode can be missing from NetworkTopology after it recovers from DEAD
> ------------------------------------------------------------------------
>
> Key: HDDS-16623
> URL: https://issues.apache.org/jira/browse/HDDS-16623
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> SCM removes a datanode from its NetworkTopology when the datanode becomes
> DEAD, and adds it back when the datanode starts heartbeating again and moves
> to HEALTHY_READONLY. Since HDDS-5437 this has been done by two event
> handlers: DeadNodeHandler removes the node and HealthyReadOnlyNodeHandler
> adds it back. The handlers run on separate EventQueue threads, and each one
> acts on a node state it read a little earlier.
> HDDS-14834 made DeadNodeHandler check the node state again just before
> removing the node. The check and the removal are still separate steps,
> though, so this can happen:
> 1. DeadNodeHandler checks the node and sees it is still DEAD.
> 2. The datanode heartbeats again. SCM moves it to HEALTHY_READONLY and fires
> HEALTHY_READONLY_NODE.
> 3. HealthyReadOnlyNodeHandler adds the node to the topology. Nothing changes,
> because the node is still there.
> 4. DeadNodeHandler removes the node.
> The datanode is now HEALTHY_READONLY, and soon HEALTHY, but it is missing
> from the topology. Placement policies that pick nodes from the topology won't
> choose it until it next moves to HEALTHY_READONLY.
> *Proposed fix*: update the topology in NodeStateManager at the same time as
> the health state changes. Remove the node when it becomes DEAD, and add it
> back when it recovers. Both changes happen inside the synchronized health
> check, so the topology always changes in the same order as the node's health.
> The event handlers no
> longer need to touch it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
