Chu Cheng Li created HDDS-17313:
-----------------------------------
Summary: SCM drops a datanode's replicas if maintenance ends right
after the datanode registers again
Key: HDDS-17313
URL: https://issues.apache.org/jira/browse/HDDS-17313
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
*Problem*
A datanode in {{IN_MAINTENANCE}} is usually shut down. SCM then marks it DEAD.
{{DeadNodeHandler}} keeps its replicas, because the datanode is in maintenance.
When the datanode starts again, it registers with SCM. If its address, ports
and version did not change, {{SCMNodeManager#register}} does not change its
health. Thus, the datanode stays DEAD until the next health check sees its
heartbeat. The health check runs every {{ozone.scm.heartbeat.thread.interval}},
which is 3 seconds by default.
Maintenance can end in this window. For example, the admin recommissions the
datanode, or the maintenance time expires. {{DatanodeAdminMonitorImpl}} then
sets the datanode to {{IN_SERVICE}}.
{{NodeStateManager#setNodeOperationalState}} fires the event for the current
health of the datanode. This health is still DEAD. Thus, SCM fires
{{DEAD_NODE}} for a datanode that is up again.
{{DeadNodeHandler}} now handles the datanode as a dead datanode that is not in
maintenance. It removes the replicas of all containers in the container list of
the datanode. These include the replicas that the datanode reported when it
registered.
The next health check marks the datanode HEALTHY. But SCM does not know its
replicas again until the next full container report. By default, this report
comes every {{hdds.container.report.interval}}, which is 60 minutes.
*Effect*
* {{ReplicationManager}} sees the containers of the datanode as
under-replicated and makes new copies. When the next full container report
comes, these containers are over-replicated, and SCM deletes the extra copies.
* SCM completes a block delete transaction when all replicas that it knows have
done the delete. The removed replica is not in this set. Thus, the replica can
keep blocks that SCM has deleted. The SCM simulation saw this.
*How it was found*
The SCM simulation from HDDS-16627 found this bug. At the end of maintenance,
the simulation restarts the datanode and recommissions it at the same time. An
operator who restarts a datanode and recommissions it within a few seconds does
the same.
*Possible fix*
When a DEAD datanode registers again, SCM can update its health at once.
Another option: {{setNodeOperationalState}} does not fire {{DEAD_NODE}} for a
datanode with a recent heartbeat.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
