I figured out what caused the slow balancing. Starting the balancer with
a too small threshold will decrease the speed dramatically:
./start-balancer.sh -threshold 0.01
2011-05-05 17:17:04,132 INFO
org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 1.26 GBbytes
in this iteration
2011-05-05 17:17:36,684 INFO
org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 1.26 GBbytes
in this iteration
2011-05-05 17:18:09,737 INFO
org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 1.26 GBbytes
in this iteration
2011-05-05 17:18:41,977 INFO
org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 1.26 GBbytes
in this iteration
as opposed to:
./start-balancer.sh
2011-05-05 17:19:01,676 INFO
org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 40 GBbytes in
this iteration
2011-05-05 17:21:36,800 INFO
org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GBbytes in
this iteration
2011-05-05 17:24:13,191 INFO
org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GBbytes in
this iteration
I'd expect setting the granularity would not affect speed, just the
stopping threshold. Perhaps a bug?
On 05/05/2011 03:43 PM, Ferdy Galema wrote:
The decommissioning was performed with solely refreshNodes, but that's
somewhat irrelevant because the balancing tests were performed after I
re-added the 11 empty nodes. (FYI the drives were formatted with
another unix fs). Though I did notice that the decommissioning shows
about the same metrics as that of the balancer test afterwards, not
very fast that is.
On 05/05/2011 02:57 PM, Mathias Herberts wrote:
Did you explicitely start a balancer or did you decommission the nodes
using dfs.hosts.exclude and a dfsadmin -refreshNodes?
On Thu, May 5, 2011 at 14:30, Ferdy Galema<ferdy.gal...@kalooga.com>
wrote:
Hi,
On our 15node cluster (1GB ethernet and 4x1TB disk per node) I
noticed that
distcp does a much better job at rebalancing than the dedicated
balancer
does. We needed to decommision 11 nodes, so that prior to
rebalancing we had
4 used and 11 empty nodes. The 4 used nodes had about 25% usage
each. Most
of our files are of average size: We have about 500K files in 280K
blocks
and 800K blocks total (blocksize is 64MB).
So I changed dfs.balance.bandwidthPerSec to 800100100 and restarted the
cluster. Started the balancer tool and I noticed that the it moved
about
200GB in 1 hour. (I grepped the balancer log for "Need to move").
After stopping the balancer I started a distcp. This tool copied
900GB in
just 45 minutes, with an average replication of 2 so it's total
throughput
was around 2.4 TB/hour. Fair enough, it is not purely rebalancing
because
the 4 overused nodes also get new blocks, still it performs much
better.
Munin confirms the much higher disk/ethernet throughputs of the distcp.
Are these characteristics to be expected? Either way, can the
balancer be
boosted even more? (Aside the dfs.balance.bandwidthPerSec property).
Ferdy.
