[
https://issues.apache.org/jira/browse/HDFS-15636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215232#comment-17215232
]
Jinglun commented on HDFS-15636:
--------------------------------
Hi [~ayushtkn], seems related, I'll give it a check.
> NameNode computes load by group when choosing datanodes.
> --------------------------------------------------------
>
> Key: HDFS-15636
> URL: https://issues.apache.org/jira/browse/HDFS-15636
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
>
> We have an HDFS cluster used for HBase with 251 ssd datanodes and 30 hdd
> datanodes. The HOT files are stored with ALL_SSD and cold ones are stored
> with HOT. There is a big chance the NameNode couldn't choose enough nodes for
> writing disk files(with storage policy HOT) because of 'NODE_TOO_BUSY'. A
> temporary solution is to increase the
> 'dfs.namenode.redundancy.considerLoad.factor'. But that may cause the
> unbalance of load of all the datanodes.
> We should let the NameNode compute load by group. The ssd nodes and hdd nodes
> are computed separately and each group has its own average load. When the
> NameNode chooses a hdd node it only compares the node's load with
> the average load of the hdd group.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
