[ https://issues.apache.org/jira/browse/HDFS-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196482#comment-15196482 ]
John Zhuge commented on HDFS-9940: ---------------------------------- Balancer uses config {{dfs.datanode.balance.max.concurrent.moves}} to set field {{Dispatcher$maxConcurrentMovesPerNode}} that sets the size of the thread pool {{moveExecutor}} local to {{Dispatcher$executePendingMove}}. * If the value is higher than {{dfs.datanode.balance.max.concurrent.moves}} on the Datanode, Balancer may send more requests than DN can handle; DN will log "Not able to copy block ... because threads quota is exceeded" and return ERROR to Balancer. Thus some Balancer threads are wasted. * if the value is smaller, the potential of the DN is not reached. I can understand the original author's decision to use the same config name. How about {{dfs.balancer.max.concurrent.moves.per.datanode}}? > Rename dfs.balancer.max.concurrent.moves to avoid confusion > ----------------------------------------------------------- > > Key: HDFS-9940 > URL: https://issues.apache.org/jira/browse/HDFS-9940 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover > Affects Versions: 2.6.0 > Reporter: John Zhuge > Assignee: John Zhuge > Priority: Minor > Labels: supportability > Fix For: 2.8.0 > > > It is very confusing for both Balancer and Datanode to use the same property > {{dfs.datanode.balance.max.concurrent.moves}}. It is especially so for the > Balancer because the property has "datanode" in the name string. Many > customers forget to set the property for the Balancer. > Change the Balancer to use a new property > {{dfs.balancer.max.concurrent.moves}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)