[ https://issues.apache.org/jira/browse/HDFS-10289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241396#comment-15241396 ]
John Zhuge commented on HDFS-10289: ----------------------------------- Thanks [~anu], I will take a look. > Balancer configures DNs directly > -------------------------------- > > Key: HDFS-10289 > URL: https://issues.apache.org/jira/browse/HDFS-10289 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover > Affects Versions: 2.6.0 > Reporter: John Zhuge > Assignee: John Zhuge > Priority: Critical > > Balancer directly configures the 2 balance-related properties > (bandwidthPerSec and concurrentMoves) on the DNs involved. > Details: > * Before each balancing iteration, set the properties on all DNs involved in > the current iteration. > * The DN property changes will not survive restart. > * Balancer gets the property values from command line or its config file. > * Need new DN APIs to query and set the 2 properties. > * No need to edit the config file on each DN or run {{hdfs dfsadmin > -setBalancerBandwidth}} to configure every DN in the cluster. > Pros: > * Improve ease of use because all configurations are done at one place, the > balancer. We saw many customers often forgot to set concurrentMoves properly > since it is required on both DN and Balancer. > * Support new DNs added between iterations > * Handle DN restarts between iterations > * May be able to dynamically adjust the thresholds in different iterations. > Don't know how useful though. > Cons: > * New DN property API > * A malicious/misconfigured balancer may overwhelm DNs. {{hdfs dfsadmin > -setBalancerBandwidth}} has the same issue. Also Balancer can only be run by > admin. > Questions: > * Can we create {{BalancerConcurrentMovesCommand}} similar to > {{BalancerBandwidthCommand}}? Can Balancer use them directly without going > through NN? > One proposal to implement HDFS-7466 calls for an API to query DN properties. > DN Conf Servlet returns all config properties. It does not return individual > property and it does not return the value set by {{hdfs dfsadmin > -setBalancerBandwidth}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)