[
https://issues.apache.org/jira/browse/HDFS-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083131#comment-14083131
]
Ming Ma commented on HDFS-6425:
-------------------------------
PostponedMisreplicatedBlocks > 1M can cause BR latency to spike to couple
seconds.
1. Given all the DNs are marked as blockContentsStale after NN fail overs,
rescanPostponedMisreplicatedBlocks isn't going to find many blocks to remove
until majority of DNs send their blockreports.
2. Normally it is ok to not remove stay over replicated right away. So
rescanPostponedMisreplicatedBlocks can wait until most if not all of the DN
storages aren't marked as blockContentsStale anymore.
Ideas on how to fix it:
Rescan postponed blocks only after all/most of DN storages aren't marked as
blockContentsStale anymore. In that way, postponed blocks won't impact BR until
most of DNs have sent BRs. After that, postponed blocks will be drained
steadily. We can do it in a background thread instead of during BR call.
Alternatively, [~lohit] and I also discussed using HashMap to store postponed
blocks, keyed by DN storage, that means each BR doesn't need to scan the whole
set and thus improve the performance.
Suggestions?
> Large postponedMisreplicatedBlocks has impact on blockReport latency
> --------------------------------------------------------------------
>
> Key: HDFS-6425
> URL: https://issues.apache.org/jira/browse/HDFS-6425
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Ming Ma
> Assignee: Ming Ma
> Attachments: HDFS-6425.patch
>
>
> Sometimes we have large number of over replicates when NN fails over. When
> the new active NN took over, over replicated blocks will be put to
> postponedMisreplicatedBlocks until all DNs for that block aren't stale
> anymore.
> We have a case where NNs flip flop. Before postponedMisreplicatedBlocks
> became empty, NN fail over again and again. So postponedMisreplicatedBlocks
> just kept increasing until the cluster is stable.
> In addition, large postponedMisreplicatedBlocks could make
> rescanPostponedMisreplicatedBlocks slow. rescanPostponedMisreplicatedBlocks
> takes write lock. So it could slow down the block report processing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
