Srikanth, The cluster is idle while balancing, and it seems to move about 2MB/minute. There is no discernable CPU load. john
From: Srikanth upputuri [mailto:srikanth.upput...@huawei.com] Sent: Thursday, September 04, 2014 12:57 AM To: user@hadoop.apache.org Subject: RE: How can I increase the speed balancing? AFAIK, this setting is meant to throttle bandwidth usage by balancer so that the balancing traffic will not severely impact the performance of the running jobs. Increasing this value will show effect only when there is enough total available bandwidth on the network. On an already overloaded network changing this value may not show much improvement. I suggest you look at the total network capacity, network usage on your datanodes to assess if there is sufficient room to increase your balancer bandwidth. Can you also try running balancer when there is no other traffic and see if changing this value has any impact. Correction: Default balancer bandwidth is 1MB/s, not 1KB/s as I mentioned in my previous post. Sorry for the typo. From: John Lilley [mailto:john.lil...@redpoint.net] Sent: 03 September 2014 17:38 To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: RE: How can I increase the speed balancing? I have also found that neither dfsadmin - setBalanacerBandwidth nor dfs.datanode.balance.bandwidthPerSec’ have any notable effect on apparent balancer rate. This is on Hadoop 2.2.0 john From: cho ju il [mailto:tjst...@kgrid.co.kr] Sent: Wednesday, September 03, 2014 12:55 AM To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: RE: How can I increase the speed balancing? Bandwidth is enough And i use command "bin/hdfs dfsadmin -setBalancerBandwidth 52428800 ". Yet balancing slow. I think because the file transfer speed is slow and move 5 files per 1 server . -----Original Message----- From: "Srikanth upputuri"<srikanth.upput...@huawei.com<mailto:srikanth.upput...@huawei.com>> To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"<user@hadoop.apache.org<mailto:user@hadoop.apache.org>>; Cc: Sent: 2014-09-03 (수) 14:10:24 Subject: RE: How can I increase the speed balancing? I am not sure what you meant by ‘Bandwidth is not a lack of data nodes’ but have you configured the balancer bandwidth property ‘dfs.datanode.balance.bandwidthPerSec’? If not it defaults to 1KB/s. You can increase this to improve the balancer speed. You may also set it dynamically using the command ‘dfsadmin -setBalanacerBandwidth newbandwidth’ before running balancer. From: cho ju il [mailto:tjst...@kgrid.co.kr] Sent: 03 September 2014 06:31 To: user@hadoop.apache.org<mailto:user@hadoop.apache.org> Subject: How can I increase the speed balancing? hadoop version 2.4.1 Balancing speed is slow. I think because the file transfer speed is slow. Bandwidth is not a lack of data nodes. How can I increase the speed balancing? *** balancer log 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.207:40010 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.205:40010 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.203:40010 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.210:40010 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.211:40010 2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.114:40010 2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.206:40010 2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.201:40010 2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.202:40010 2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.204:40010 2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 over-utilized: [Source[192.168.0.203:40010, utilization=99.99693907952093], Source[192.168.0.201:40010, utilization=99.99713240471648], Source[192.168.0.202:40010, utilization=99.99652052169367]] 2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 underutilized: [BalancerDatanode[192.168.0.211:40010, utilization=62.735024524531006], BalancerDatanode[192.168.0.114:40010, utilization=2.3174560700459224]] 2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 70.30 TB to make the cluster balanced. 2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.0.203:40010 to 192.168.0.211:40010 2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.0.201:40010 to 192.168.0.114:40010 2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 20 GB in this iteration 2014-09-03 09:44:23,643 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1077216256_3475617 with size=1746577 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.205:40010 2014-09-03 09:45:38,730 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1076170742_2430103 with size=16746627 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.203:40010 2014-09-03 09:46:42,649 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1077216315_3475676 with size=38338949 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.206:40010 2014-09-03 09:50:13,239 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1074229029_488207 with size=89080307 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.203:40010 2014-09-03 09:50:58,659 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1077216317_3475678 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.210:40010 2014-09-03 09:51:15,660 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1075275002_1534363 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.204:40010 2014-09-03 09:51:19,661 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1077216257_3475618 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.202:40010 2014-09-03 09:51:54,243 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1076170740_2430101 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.203:40010 2014-09-03 09:52:15,242 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1074229048_488226 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.205:40010 2014-09-03 09:52:22,744 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1076170741_2430102 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.203:40010 [Image removed by sender.] [Image removed by sender.]
