[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907769#comment-14907769 ]
Masatake Iwasaki commented on HDFS-1172: ---------------------------------------- bq. Looks like the rebased patches only put not-yet-received replicas into the pending replication queue, but BlockManager#checkReplication has not been updated accordingly I think it is better to leave {{BlockManager#checkReplication}} as is here. Though it may add block having pending replicas to {{neededReplications}}, the replication will not be scheduled as far as the replica is in {{pendingReplications}} because {{BlockManager#hasEnoughEffectiveReplicas}} takes it into account. {{BlockManager#isNeededReplication}} is used in other places. Keeping the condition for updating {{neededReplications}} consistent makes the code clear and will avoid potential bugs. bq. 2. makes sure that blocks except for the last block in a file under-construction get replicated when under-replicated; This will allow a decommissioning datanode to finish decommissioning even it has replicas in files under construction. {{TestReplication#testReplicationWhileUnderConstruction}} checks this is satisfied. > Blocks in newly completed files are considered under-replicated too quickly > --------------------------------------------------------------------------- > > Key: HDFS-1172 > URL: https://issues.apache.org/jira/browse/HDFS-1172 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode > Affects Versions: 0.21.0 > Reporter: Todd Lipcon > Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch, > HDFS-1172.009.patch, HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt, > replicateBlocksFUC.patch, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch > > > I've seen this for a long time, and imagine it's a known issue, but couldn't > find an existing JIRA. It often happens that we see the NN schedule > replication on the last block of files very quickly after they're completed, > before the other DNs in the pipeline have a chance to report the new block. > This results in a lot of extra replication work on the cluster, as we > replicate the block and then end up with multiple excess replicas which are > very quickly deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)