[ https://issues.apache.org/jira/browse/HDFS-15636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215231#comment-17215231 ]
Jinglun commented on HDFS-15636: -------------------------------- We can implement this by 2 steps: # Let NameNode support computing load with group. The NameNode should resolve group for each datanode and count 'nodes in service' and 'xceiver count' for each group(in DatanodeStats). # Add a new BlockPlacementPolicy which considers load with group. > NameNode computes load by group when choosing datanodes. > -------------------------------------------------------- > > Key: HDFS-15636 > URL: https://issues.apache.org/jira/browse/HDFS-15636 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Jinglun > Assignee: Jinglun > Priority: Major > > We have an HDFS cluster used for HBase with 251 ssd datanodes and 30 hdd > datanodes. The HOT files are stored with ALL_SSD and cold ones are stored > with HOT. There is a big chance the NameNode couldn't choose enough nodes for > writing disk files(with storage policy HOT) because of 'NODE_TOO_BUSY'. A > temporary solution isĀ to increase the > 'dfs.namenode.redundancy.considerLoad.factor'. But that may cause the > unbalance of load of all the datanodes. > We should let the NameNode compute load by group. The ssd nodes and hdd nodes > are computed separately and each group has its own average load. When the > NameNode chooses a hdd node it only compares the node's load with > the average load of the hdd group. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org