[ https://issues.apache.org/jira/browse/HDFS-14383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215242#comment-17215242 ]
Jinglun commented on HDFS-14383: -------------------------------- I met the same problem recently. This patch makes sense to me. Thanks [~ayushtkn] your working. > Compute datanode load based on StoragePolicy > -------------------------------------------- > > Key: HDFS-14383 > URL: https://issues.apache.org/jira/browse/HDFS-14383 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs, namenode > Affects Versions: 2.7.3, 3.1.2 > Reporter: Karthik Palanisamy > Assignee: Ayush Saxena > Priority: Major > Attachments: HDFS-14383-01.patch, HDFS-14383-02.patch > > > Datanode load check logic needs to be changed because existing computation > will not consider StoragePolicy. > DatanodeManager#getInServiceXceiverAverage > {code} > public double getInServiceXceiverAverage() { > double avgLoad = 0; > final int nodes = getNumDatanodesInService(); > if (nodes != 0) { > final int xceivers = heartbeatManager > .getInServiceXceiverCount(); > avgLoad = (double)xceivers/nodes; > } > return avgLoad; > } > {code} > > For example: with 10 nodes (HOT), average 50 xceivers and 90 nodes (COLD) > with average 10 xceivers the calculated threshold by the NN is 28 (((500 + > 900)/100)*2), which means those 10 nodes (the whole HOT tier) becomes > unavailable when the COLD tier nodes are barely in use. Turning this check > off helps to mitigate this issue, however the > dfs.namenode.replication.considerLoad helps to "balance" the load of the DNs, > upon turning it off can lead to situations where specific DNs are > "overloaded". -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org