[ https://issues.apache.org/jira/browse/HDFS-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186007#comment-14186007 ]
Hadoop QA commented on HDFS-7235: --------------------------------- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677434/HDFS-7235.007.patch against trunk revision a16d022. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8557//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8557//console This message is automatically generated. > Can not decommission DN which has invalid block due to bad disk > --------------------------------------------------------------- > > Key: HDFS-7235 > URL: https://issues.apache.org/jira/browse/HDFS-7235 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode > Affects Versions: 2.6.0 > Reporter: Yongjun Zhang > Assignee: Yongjun Zhang > Attachments: HDFS-7235.001.patch, HDFS-7235.002.patch, > HDFS-7235.003.patch, HDFS-7235.004.patch, HDFS-7235.005.patch, > HDFS-7235.006.patch, HDFS-7235.007.patch > > > When to decommission a DN, the process hangs. > What happens is, when NN chooses a replica as a source to replicate data on > the to-be-decommissioned DN to other DNs, it favors choosing this DN > to-be-decommissioned as the source of transfer (see BlockManager.java). > However, because of the bad disk, the DN would detect the source block to be > transfered as invalidBlock with the following logic in FsDatasetImpl.java: > {code} > /** Does the block exist and have the given state? */ > private boolean isValid(final ExtendedBlock b, final ReplicaState state) { > final ReplicaInfo replicaInfo = volumeMap.get(b.getBlockPoolId(), > b.getLocalBlock()); > return replicaInfo != null > && replicaInfo.getState() == state > && replicaInfo.getBlockFile().exists(); > } > {code} > The reason that this method returns false (detecting invalid block) is > because the block file doesn't exist due to bad disk in this case. > The key issue we found here is, after DN detects an invalid block for the > above reason, it doesn't report the invalid block back to NN, thus NN doesn't > know that the block is corrupted, and keeps sending the data transfer request > to the same DN to be decommissioned, again and again. This caused an infinite > loop, so the decommission process hangs. > Thanks [~qwertymaniac] for reporting the issue and initial analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)