[ https://issues.apache.org/jira/browse/HDFS-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302299#comment-14302299 ]
Zhe Zhang commented on HDFS-7725: --------------------------------- bq. However, HDFS-7374 introduces another inconsistency when recomm is involved. The second sequence in the JIRA description looks correct? By reading the HDFS-7374 patch I see the potential issue is that {{HeartbeatManager}} is bypassed when decommissioning a dead node: {code} + if (!node.isDecommissionInProgress()) { + if (!node.isAlive) { + LOG.info("Dead node " + node + " is decommissioned immediately."); + node.setDecommissioned(); + } else if (!node.isDecommissioned()) { + for (DatanodeStorageInfo storage : node.getStorageInfos()) { + LOG.info("Start Decommissioning " + node + " " + storage + + " with " + storage.numBlocks() + " blocks"); + } + heartbeatManager.startDecommission(node); {code} It seems {{DatanodeManager}} should still route the call to {{HeartbeatManager}}, and {{HeartbeatManager#startDecommission}} should handle the dead node logic. Maybe we should wait for HDFS-7411 to be committed and revisit the change? > Incorrect "nodes in service" metrics caused all writes to fail > -------------------------------------------------------------- > > Key: HDFS-7725 > URL: https://issues.apache.org/jira/browse/HDFS-7725 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Ming Ma > Assignee: Ming Ma > Attachments: HDFS-7725.patch > > > One of our clusters sometimes couldn't allocate blocks from any DNs. > BlockPlacementPolicyDefault complains with the following messages for all DNs. > {noformat} > the node is too busy (load:x > y) > {noformat} > It turns out the {{HeartbeatManager}}'s {{nodesInService}} was computed > incorrectly when admins decomm or recomm dead nodes. Here are two scenarios. > * Decomm dead nodes. It turns out HDFS-7374 has fixed it; not sure if it is > intentional. cc / [~zhz], [~andrew.wang], [~atm] Here is the sequence of > event without HDFS-7374. > ** Cluster has one live node. nodesInService == 1 > ** The node becomes dead. nodesInService == 0 > ** Decomm the node. nodesInService == -1 > * However, HDFS-7374 introduces another inconsistency when recomm is involved. > ** Cluster has one live node. nodesInService == 1 > ** The node becomes dead. nodesInService == 0 > ** Decomm the node. nodesInService == 0 > ** Recomm the node. nodesInService == 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)