I have also found that neither dfsadmin - setBalanacerBandwidth nor dfs.datanode.balance.bandwidthPerSec’ have any notable effect on apparent balancer rate. This is on Hadoop 2.2.0
john From: cho ju il [mailto:tjst...@kgrid.co.kr] Sent: Wednesday, September 03, 2014 12:55 AM To: user@hadoop.apache.org Subject: RE: How can I increase the speed balancing? Bandwidth is enough And i use command "bin/hdfs dfsadmin -setBalancerBandwidth 52428800 ". Yet balancing slow. I think because the file transfer speed is slow and move 5 files per 1 server . -----Original Message----- From: "Srikanth upputuri"<srikanth.upput...@huawei.com> To: "user@hadoop.apache.org"<user@hadoop.apache.org>; Cc: Sent: 2014-09-03 (수) 14:10:24 Subject: RE: How can I increase the speed balancing? I am not sure what you meant by ‘Bandwidth is not a lack of data nodes’ but have you configured the balancer bandwidth property ‘dfs.datanode.balance.bandwidthPerSec’? If not it defaults to 1KB/s. You can increase this to improve the balancer speed. You may also set it dynamically using the command ‘dfsadmin -setBalanacerBandwidth newbandwidth’ before running balancer. From: cho ju il [mailto:tjst...@kgrid.co.kr] Sent: 03 September 2014 06:31 To: user@hadoop.apache.org Subject: How can I increase the speed balancing? hadoop version 2.4.1 Balancing speed is slow. I think because the file transfer speed is slow. Bandwidth is not a lack of data nodes. How can I increase the speed balancing? *** balancer log 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.207:40010 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.205:40010 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.203:40010 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.210:40010 2014-09-03 09:44:18,041 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.211:40010 2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.114:40010 2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.206:40010 2014-09-03 09:44:18,042 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.201:40010 2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.202:40010 2014-09-03 09:44:18,043 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.0.204:40010 2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 over-utilized: [Source[192.168.0.203:40010, utilization=99.99693907952093], Source[192.168.0.201:40010, utilization=99.99713240471648], Source[192.168.0.202:40010, utilization=99.99652052169367]] 2014-09-03 09:44:18,043 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 underutilized: [BalancerDatanode[192.168.0.211:40010, utilization=62.735024524531006], BalancerDatanode[192.168.0.114:40010, utilization=2.3174560700459224]] 2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 70.30 TB to make the cluster balanced. 2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.0.203:40010 to 192.168.0.211:40010 2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.0.201:40010 to 192.168.0.114:40010 2014-09-03 09:44:18,044 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 20 GB in this iteration 2014-09-03 09:44:23,643 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1077216256_3475617 with size=1746577 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.205:40010 2014-09-03 09:45:38,730 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1076170742_2430103 with size=16746627 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.203:40010 2014-09-03 09:46:42,649 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1077216315_3475676 with size=38338949 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.206:40010 2014-09-03 09:50:13,239 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1074229029_488207 with size=89080307 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.203:40010 2014-09-03 09:50:58,659 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1077216317_3475678 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.210:40010 2014-09-03 09:51:15,660 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1075275002_1534363 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.204:40010 2014-09-03 09:51:19,661 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1077216257_3475618 with size=134217728 from 192.168.0.201:40010 to 192.168.0.114:40010 through 192.168.0.202:40010 2014-09-03 09:51:54,243 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1076170740_2430101 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.203:40010 2014-09-03 09:52:15,242 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1074229048_488226 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.205:40010 2014-09-03 09:52:22,744 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1076170741_2430102 with size=134217728 from 192.168.0.203:40010 to 192.168.0.211:40010 through 192.168.0.203:40010 [Image removed by sender.] [Image removed by sender.]
