[
https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15949905#comment-15949905
]
Konstantin Shvachko commented on HDFS-11384:
--------------------------------------------
Combining my and Vinitha's proposals in a common list here, for further
discussion.
# Add a parameter to {{getBlockList(long delay)}}, where {{delay}} is a random
time within a reasonable interval, which Balancer should wait for before
sending the {{getBlocks()}} RPC to NN. The delay is only applied once, and set
to 0 once applied. This looks rather straightforward to me.
# Allocate a reasonable throughput of {{getBlocks()}} RPCs to NN, and delay
calls if the quota is exceeded. This is similar to [~benoyantony]'s proposal,
but allows to precisely specify how much of NN RPC bandwidth is allocated for
the Balancer.
# Dispatcher can get the blocks directly from the particular DN instead of NN.
This makes {{getBlocksList()}} a distributed operation and doesn't impact any
specific node.
# Dispatcher can fetch the blocks from the Standby NN instead of the active.
Balancer should be able to tolerate reasonable degree of staleness.
I think (1) is quite easy and can solve immediate problem with bursting RPCs.
I really like Vinitha's (3) obtaining block info directly from DNs. I did not
look how much harder it is than other approaches.
(2) and (4) are complimentary, that is we can have a limit on Balancer RPC
bandwidth to NN, but run it against SBN.
> Add option for balancer to disperse getBlocks calls to avoid NameNode's
> rpc.CallQueueLength spike
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-11384
> URL: https://issues.apache.org/jira/browse/HDFS-11384
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
> Affects Versions: 2.7.3
> Reporter: yunjiong zhao
> Assignee: yunjiong zhao
> Attachments: balancer.day.png, balancer.week.png,
> HDFS-11384.001.patch, HDFS-11384.002.patch
>
>
> When running balancer on hadoop cluster which have more than 3000 Datanodes
> will cause NameNode's rpc.CallQueueLength spike. We observed this situation
> could cause Hbase cluster failure due to RegionServer's WAL timeout.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
