Jinglun created HDFS-15636:
------------------------------
Summary: NameNode computes load by group when choosing datanodes.
Key: HDFS-15636
URL: https://issues.apache.org/jira/browse/HDFS-15636
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Jinglun
We have an HDFS cluster used for HBase with 251 ssd datanodes and 30 hdd
datanodes. The HOT files are stored with ALL_SSD and cold ones are stored with
HOT. There is a big chance the NameNode couldn't choose enough nodes for
writing disk files(with storage policy HOT) because of 'NODE_TOO_BUSY'. A
temporary solution is to increase the
'dfs.namenode.redundancy.considerLoad.factor'. But that may cause the unbalance
of load of all the datanodes.
We should let the NameNode compute load by group. The ssd nodes and hdd nodes
are computed separately and each group has its own average load. When the
NameNode chooses a hdd node it only compares the node's load with
the average load of the hdd group.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
