[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[ https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10297: --- Resolution: Fixed Fix Version/s: (was: 2.9.0) 2.8.0 Status: Resolved (was: Patch Available) Pushed, thanks John! > Increase default balance bandwidth and concurrent moves > --- > > Key: HDFS-10297 > URL: https://issues.apache.org/jira/browse/HDFS-10297 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Labels: supportability > Fix For: 2.8.0 > > Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch, > HDFS-10297.003.branch-2.8.patch, HDFS-10297.003.patch > > > Adjust the default values to better support the current level of customer > host and network configurations. > Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} > from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network. > Increase the default for property > {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and > Balancer. The default number of DN receiver threads is 4096. The default > number of balancer mover threads is 1000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[ https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-10297: -- Attachment: HDFS-10297.003.branch-2.8.patch Upload HDFS-10297.003.branch-2.8.patch for branch 2.8. > Increase default balance bandwidth and concurrent moves > --- > > Key: HDFS-10297 > URL: https://issues.apache.org/jira/browse/HDFS-10297 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Labels: supportability > Fix For: 2.9.0 > > Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch, > HDFS-10297.003.branch-2.8.patch, HDFS-10297.003.patch > > > Adjust the default values to better support the current level of customer > host and network configurations. > Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} > from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network. > Increase the default for property > {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and > Balancer. The default number of DN receiver threads is 4096. The default > number of balancer mover threads is 1000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[ https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10297: --- Fix Version/s: 2.9.0 Committed to trunk and branch-2, branch-2.8 wasn't clean. John, if you want it in 2.8, do you mind providing a branch-2.8 patch? Else we can just resolve. > Increase default balance bandwidth and concurrent moves > --- > > Key: HDFS-10297 > URL: https://issues.apache.org/jira/browse/HDFS-10297 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Labels: supportability > Fix For: 2.9.0 > > Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch, > HDFS-10297.003.patch > > > Adjust the default values to better support the current level of customer > host and network configurations. > Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} > from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network. > Increase the default for property > {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and > Balancer. The default number of DN receiver threads is 4096. The default > number of balancer mover threads is 1000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[ https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-10297: -- Attachment: HDFS-10297.003.patch Patch 003: * Fix checkstyle > Increase default balance bandwidth and concurrent moves > --- > > Key: HDFS-10297 > URL: https://issues.apache.org/jira/browse/HDFS-10297 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Labels: supportability > Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch, > HDFS-10297.003.patch > > > Adjust the default values to better support the current level of customer > host and network configurations. > Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} > from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network. > Increase the default for property > {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and > Balancer. The default number of DN receiver threads is 4096. The default > number of balancer mover threads is 1000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[ https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-10297: -- Attachment: HDFS-10297.002.patch Patch 002: * Fix checkstyle errors > Increase default balance bandwidth and concurrent moves > --- > > Key: HDFS-10297 > URL: https://issues.apache.org/jira/browse/HDFS-10297 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Labels: supportability > Attachments: HDFS-10297.001.patch, HDFS-10297.002.patch > > > Adjust the default values to better support the current level of customer > host and network configurations. > Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} > from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network. > Increase the default for property > {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and > Balancer. The default number of DN receiver threads is 4096. The default > number of balancer mover threads is 1000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[ https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-10297: - Fix Version/s: (was: 2.8.0) > Increase default balance bandwidth and concurrent moves > --- > > Key: HDFS-10297 > URL: https://issues.apache.org/jira/browse/HDFS-10297 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Labels: supportability > Attachments: HDFS-10297.001.patch > > > Adjust the default values to better support the current level of customer > host and network configurations. > Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} > from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network. > Increase the default for property > {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and > Balancer. The default number of DN receiver threads is 4096. The default > number of balancer mover threads is 1000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[ https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-10297: -- Fix Version/s: 2.8.0 Status: Patch Available (was: Open) > Increase default balance bandwidth and concurrent moves > --- > > Key: HDFS-10297 > URL: https://issues.apache.org/jira/browse/HDFS-10297 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Labels: supportability > Fix For: 2.8.0 > > Attachments: HDFS-10297.001.patch > > > Adjust the default values to better support the current level of customer > host and network configurations. > Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} > from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network. > Increase the default for property > {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and > Balancer. The default number of DN receiver threads is 4096. The default > number of balancer mover threads is 1000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-10297) Increase default balance bandwidth and concurrent moves
[ https://issues.apache.org/jira/browse/HDFS-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-10297: -- Attachment: HDFS-10297.001.patch Patch 001: * Change values in {{hdfs-default.xml}} * Change values of {{DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_DEFAULT}} and {{DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT}} Test output: {noformat} $ ( cd hadoop-hdfs-project/hadoop-hdfs ; mvn test -Dtest=TestBalancer#testBalancer2 ) $ grep 'concurrent\.moves\|Number threads for balancing' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output* 2016-04-15 10:46:58,563 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:(79)) - Number threads for balancing is 50 2016-04-15 10:46:58,781 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:(79)) - Number threads for balancing is 50 2016-04-15 10:46:59,942 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:(79)) - Number threads for balancing is 50 2016-04-15 10:47:00,153 [Thread-0] INFO balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50) 2016-04-15 10:47:04,205 [Thread-0] INFO balancer.Balancer (Balancer.java:getInt(240)) - dfs.datanode.balance.max.concurrent.moves = 50 (default=50) [jzhuge@jzhuge-MBP hadoop](trunk *)$ grep 'bandwidthPer\|Balancing bandwidth' hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports/*output* 2016-04-15 10:46:58,563 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:(78)) - Balancing bandwidth is 10485760 bytes/s 2016-04-15 10:46:58,781 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:(78)) - Balancing bandwidth is 10485760 bytes/s 2016-04-15 10:46:59,942 [Thread-0] INFO datanode.DataNode (DataXceiverServer.java:(78)) - Balancing bandwidth is 10485760 bytes/s {noformat} > Increase default balance bandwidth and concurrent moves > --- > > Key: HDFS-10297 > URL: https://issues.apache.org/jira/browse/HDFS-10297 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Labels: supportability > Attachments: HDFS-10297.001.patch > > > Adjust the default values to better support the current level of customer > host and network configurations. > Increase the default for property {{dfs.datanode.balance.bandwidthPerSec}} > from 1 to 10 MB. Apply to DN. 10 MB/s is about 10% of the GbE network. > Increase the default for property > {{dfs.datanode.balance.max.concurrent.moves}} from 5 to 50. Apply to DN and > Balancer. The default number of DN receiver threads is 4096. The default > number of balancer mover threads is 1000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)