[ https://issues.apache.org/jira/browse/HDFS-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ming Ma updated HDFS-6791: -------------------------- Attachment: HDFS-6791-3.patch Updated patch to make the test more robust. After the node is considered dead by DatanodeManager, make sure all blocks have been removed for that node and BlockManager gets a chance to update decommission state. After all that, the test will verify the decommission status of the node. > A block could remain under replicated if all of its replicas are on > decommissioned nodes > ---------------------------------------------------------------------------------------- > > Key: HDFS-6791 > URL: https://issues.apache.org/jira/browse/HDFS-6791 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Ming Ma > Assignee: Ming Ma > Attachments: HDFS-6791-2.patch, HDFS-6791-3.patch, HDFS-6791.patch > > > Here is the scenario. > 1. Normally before NN transitions a DN to decommissioned state, enough > replicas have been copied to other "in service" DNs. However, in some rare > situations, the cluster got into a state where a DN is in decommissioned > state and a block's only replica is on that DN. In such state, the number of > replication reported by fsck is 1; the block just stays in under replicated > state; applications can still read the data, given decommissioned node can > served read traffic. > This can happen in some error situations such DN failure or NN failover. For > example > a) a block's only replica is node A temporarily. > b) Start decommission process on node A. > c) When node A is in "decommission-in-progress" state, node A crashed. NN > will mark node A as dead. > d) After node A rejoins the cluster, NN will mark node A as decommissioned. > 2. In theory, NN should take care of under replicated blocks. But it doesn't > for this special case where the only replica is on decommissioned node. That > is because NN has the policy of "decommissioned node can't be picked the > source node for replication". > {noformat} > BlockManager.java > chooseSourceDatanode > // never use already decommissioned nodes > if(node.isDecommissioned()) > continue; > {noformat} > 3. Given NN marks the node as decommissioned, admins will shutdown the > datanode. Under replicated blocks turn into missing blocks. > 4. The workaround is to recommission the node so that NN can start the > replication from the node. -- This message was sent by Atlassian JIRA (v6.2#6252)