[ https://issues.apache.org/jira/browse/HDFS-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215288#comment-17215288 ]
Stephen O'Donnell commented on HDFS-15634: ------------------------------------------ {quote} Proposal: Invalidate these blocks once they are replicated and there are enough live replicas in the cluster. {quote} Looking at the PR, you are adding these blocks to addToInvalidates(...) which will actually remove the replicas from the DNs. I am not sure this is a good idea, for a few reasons: 1. Right now, a decommissioned DN is untouched by the process - if something goes wrong with decommission (which we have seen happen) we can just recommission the node again and know all the blocks are still safely present. 2. I seem to recall there are some edge cases where a decommissioned but still online replica can be read. 3. On some clusters, nodes are decommissioned for maintenance (yes, they should use maintenance mode, but some don't) such as OS upgrades and then recommissioned. In these cases, when the DN rejoins, the blocks will become over replicated and then the NN will remove replicas randomly. This is arguably better than adding back an empty node, which may require running the balancer to move data onto it. If we remove the blocks from the DN while it is decommissioning, then on recommission we can only ever add back an empty node. {quote} A recent shutdown of decommissioned datanodes to finished the flow caused Namenode latency spike since namenode needs to remove all of the blocks from its memory and this step requires holding write lock. If we have gradually invalidated these blocks the deletion will be much easier and faster. {quote} What version were you running when you saw this problem? How many blocks approximately were on the DNs which were stopped after decommission completed? How many decommissioned hosts were stopped when this happened? I am wondering if there would be a better way to handle this, possibly yielding the write lock which removing the blocks periodically, as this same problem would exist for a node going dead unexpectedly and not just during decommission. > Invalidate block on decommissioning DataNode after replication > -------------------------------------------------------------- > > Key: HDFS-15634 > URL: https://issues.apache.org/jira/browse/HDFS-15634 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs > Reporter: Fengnan Li > Assignee: Fengnan Li > Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Right now when a DataNode starts decommission, Namenode will mark it as > decommissioning and its blocks will be replicated over to different > DataNodes, then marked as decommissioned. These blocks are not touched since > they are not counted as live replicas. > Proposal: Invalidate these blocks once they are replicated and there are > enough live replicas in the cluster. > Reason: A recent shutdown of decommissioned datanodes to finished the flow > caused Namenode latency spike since namenode needs to remove all of the > blocks from its memory and this step requires holding write lock. If we have > gradually invalidated these blocks the deletion will be much easier and > faster. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org