[ https://issues.apache.org/jira/browse/HDFS-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955487#comment-16955487 ]
Xiaoqiao He commented on HDFS-14882: ------------------------------------ Thanks [~ayushtkn] for your reviews. I have addressed all the latest comments, I think. {quote}You even need to add the new config in Hdfs-defaults.xml {quote} I try to reused the config which has used by BlockPlacementPolicyDefault#isGoodDatanode. IMO, they both follow the same semantic, so I think we do not need to add another one. I would like to give a brief introduction for this changes. Generally, this patch try to re-order nodes who have same topology distance to client and based on Load descend order. Firstly, calculate all distances from nodes to client. Then, try to use #start and #end to embrace nodes with the same distances. Thirdly, re-sort by load descend order. (Note: Skip it when section length is less than 2 since it is unnecessary to sort one node.) > Consider DataNode load when #getBlockLocation > --------------------------------------------- > > Key: HDFS-14882 > URL: https://issues.apache.org/jira/browse/HDFS-14882 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Reporter: Xiaoqiao He > Assignee: Xiaoqiao He > Priority: Major > Attachments: HDFS-14882.001.patch, HDFS-14882.002.patch, > HDFS-14882.003.patch > > > Currently, we consider load of datanode when #chooseTarget for writer, > however not consider it for reader. Thus, the process slot of datanode could > be occupied by #BlockSender for reader, and disk/network will be busy > workload, then meet some slow node exception. IIRC same case is reported > times. Based on the fact, I propose to consider load for reader same as it > did #chooseTarget for writer. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org