[
https://issues.apache.org/jira/browse/HDFS-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14249908#comment-14249908
]
Hudson commented on HDFS-6425:
------------------------------
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #41 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/41/])
HDFS-6425. Large postponedMisreplicatedBlocks has impact on blockReport
latency. Contributed by Ming Ma. (kihwal: rev
b7923a356e9f111619375b94d12749d634069347)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencing.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDNFencingWithReplication.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
> Large postponedMisreplicatedBlocks has impact on blockReport latency
> --------------------------------------------------------------------
>
> Key: HDFS-6425
> URL: https://issues.apache.org/jira/browse/HDFS-6425
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ming Ma
> Assignee: Ming Ma
> Fix For: 2.7.0
>
> Attachments: HDFS-6425-2.patch, HDFS-6425-3.patch,
> HDFS-6425-Test-Case.pdf, HDFS-6425.patch
>
>
> Sometimes we have large number of over replicates when NN fails over. When
> the new active NN took over, over replicated blocks will be put to
> postponedMisreplicatedBlocks until all DNs for that block aren't stale
> anymore.
> We have a case where NNs flip flop. Before postponedMisreplicatedBlocks
> became empty, NN fail over again and again. So postponedMisreplicatedBlocks
> just kept increasing until the cluster is stable.
> In addition, large postponedMisreplicatedBlocks could make
> rescanPostponedMisreplicatedBlocks slow. rescanPostponedMisreplicatedBlocks
> takes write lock. So it could slow down the block report processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
