[
https://issues.apache.org/jira/browse/HDFS-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288601#comment-13288601
]
Ashish Singhi commented on HDFS-3377:
-------------------------------------
Hi Andreina.
There is no restriction on the number of blocks a datanode can move in one
iteration.
{quote}But MAX_NUM_CONCURRENT_MOVES in one iterations is hard coded to
5.{quote}
MAX_NUM_CONCURRENT_MOVES - specifies the maximum number of concurrent block
moves for balancing purpose at a datanode can be scheduled at one single point
of time(range varies from cluster to cluster), not in one iteration.
Following are the balancer outputs obtained by executing the adove mentioned
scenario.
{code}
./hdfs balancer -threshold 1
Listening for transport dt_socket at address: 8050
12/06/04 11:38:47 INFO balancer.Balancer: Using a threshold of 1.0
12/06/04 11:38:47 INFO balancer.Balancer: namenodes =
[hdfs://HOST-10-18-40-117:9000]
12/06/04 11:38:47 INFO balancer.Balancer: p =
Balancer.Parameters[BalancingPolicy.Node, threshold=1.0]
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
Bytes Being Moved
12/06/04 11:38:51 INFO balancer.Balancer: Block token params received from NN:
keyUpdateInterval=600 min(s), tokenLifetime=600 min(s)
12/06/04 11:38:52 INFO block.BlockTokenSecretManager: Setting block keys
12/06/04 11:38:52 INFO balancer.Balancer: Balancer will update its block keys
every 150 minute(s)
12/06/04 11:38:52 INFO block.BlockTokenSecretManager: Setting block keys
12/06/04 11:38:52 INFO net.NetworkTopology: Adding a new node:
/datacenter1/rack1/xx.xx.xx.xx:50276
12/06/04 11:38:52 INFO net.NetworkTopology: Adding a new node:
/datacenter1/rack1/xx.xx.xx.xx:50176
12/06/04 11:38:52 INFO balancer.Balancer: 1 over-utilized:
[Source[xx.xx.xx.xx:50176, utilization=8.37380402300465]]
12/06/04 11:38:52 INFO balancer.Balancer: 1 underutilized:
[BalancerDatanode[xx.xx.xx.xx:50276, utilization=2.8947382057801816]]
12/06/04 11:38:52 INFO balancer.Balancer: Need to move 1.22 GB to make the
cluster balanced.
12/06/04 11:38:52 INFO balancer.Balancer: Decided to move 716.13 MB bytes from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276
12/06/04 11:38:52 INFO balancer.Balancer: Will move 716.13 MB in this iteration
May 18, 2012 11:38:52 AM 0 0 KB 1.22 GB
716.13 MB
12/06/04 11:44:06 INFO balancer.Balancer: Moving block -604085946676187269 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:44:11 INFO balancer.Balancer: Moving block -8407984470383132708
from xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is
succeeded.
12/06/04 11:44:28 INFO balancer.Balancer: Moving block -6889513481040935685
from xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is
succeeded.
12/06/04 11:44:45 INFO balancer.Balancer: Moving block 8195882447563616912 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:44:51 INFO balancer.Balancer: Moving block 8785624464350544567 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:46:55 INFO balancer.Balancer: Moving block 2826762635631555809 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:49:55 INFO balancer.Balancer: Moving block -8045953083201474660
from xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is
succeeded.
12/06/04 11:50:24 INFO balancer.Balancer: Moving block 7717431073633882015 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:50:27 INFO balancer.Balancer: Moving block 6479080974771358747 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:51:15 INFO balancer.Balancer: Moving block -8899955335678387087
from xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is
succeeded.
12/06/04 11:51:49 INFO balancer.Balancer: Moving block 1070038993680508333 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:52:02 INFO net.NetworkTopology: Adding a new node:
/datacenter1/rack1/xx.xx.xx.xx:50176
12/06/04 11:52:02 INFO net.NetworkTopology: Adding a new node:
/datacenter1/rack1/xx.xx.xx.xx:50276
12/06/04 11:52:02 INFO balancer.Balancer: 1 over-utilized:
[Source[xx.xx.xx.xx:50176, utilization=7.332913449391756]]
12/06/04 11:52:02 INFO balancer.Balancer: 1 underutilized:
[BalancerDatanode[xx.xx.xx.xx:50276, utilization=4.231341505676603]]
12/06/04 11:52:02 INFO balancer.Balancer: Need to move 394.43 MB to make the
cluster balanced.
12/06/04 11:52:02 INFO balancer.Balancer: Decided to move 716.13 MB bytes from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276
12/06/04 11:52:02 INFO balancer.Balancer: Will move 716.13 MB in this iteration
May 18, 2012 11:52:02 AM 1 0 KB 394.43 MB
716.13 MB
12/06/04 11:57:25 INFO balancer.Balancer: Moving block -5591766437649317399
from xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is
succeeded.
12/06/04 11:57:42 INFO balancer.Balancer: Moving block 9198575182337433559 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:57:56 INFO balancer.Balancer: Moving block 6855524568701374882 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:58:02 INFO balancer.Balancer: Moving block 4747385924662820878 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 11:58:29 INFO balancer.Balancer: Moving block -5973238543364868278
from xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is
succeeded.
12/06/04 12:02:58 INFO balancer.Balancer: Moving block 5717753712470906112 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 12:03:06 INFO balancer.Balancer: Moving block -3396403038672813493
from xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is
succeeded.
12/06/04 12:03:42 INFO balancer.Balancer: Moving block 175510590632581188 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 12:03:48 INFO balancer.Balancer: Moving block 735609460427544500 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 12:04:01 INFO balancer.Balancer: Moving block 9011172593322217078 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 12:05:42 INFO balancer.Balancer: Moving block 3659606766577555250 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 12:05:44 INFO balancer.Balancer: Moving block 3186378020600152947 from
xx.xx.xx.xx:50176 to xx.xx.xx.xx:50276 through xx.xx.xx.xx:50176 is succeeded.
12/06/04 12:06:12 INFO net.NetworkTopology: Adding a new node:
/datacenter1/rack1/xx.xx.xx.xx:50176
12/06/04 12:06:12 INFO net.NetworkTopology: Adding a new node:
/datacenter1/rack1/xx.xx.xx.xx:50276
12/06/04 12:06:12 INFO balancer.Balancer: 0 over-utilized: []
12/06/04 12:06:12 INFO balancer.Balancer: 0 underutilized: []
The cluster is balanced. Exiting...
12/06/04 12:06:13 INFO balancer.Balancer: InterruptedException in block key
updater thread
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hdfs.server.balancer.NameNodeConnector$BlockKeyUpdater.run(NameNodeConnector.java:200)
at java.lang.Thread.run(Thread.java:619)
Balancing took 27.499833333333335 minutes
{code}
Here you can observe that,
In the first iteration, the first 5 blocks had moved between 11:44:06 -
11:44:51
and the next 5 blocks had been moved between 11:49:55 - 11:51:49.
So at one single point of time (here the range is one minute) there are no more
than 5 blocks movement from the datanode.
Conclusion - Balancer can move more than 5 blocks from one datanode in one
iteration but not at a single point of time.
> While Balancing more than 10 Blocks are being moved from one DN even though
> the maximum number of blocks to be moved in an iterations is hard coded to 5
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3377
> URL: https://issues.apache.org/jira/browse/HDFS-3377
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer
> Affects Versions: 2.0.0-alpha
> Reporter: J.Andreina
>
> Replication factor= 1,block size is default value
> Step 1: Start NN,DN1
> Step 2: Pump 5 GB of data.
> Step 3: Start DN2 and issue balancer with threshold value 1
> In the balancer report and the NN logs displays that more than 8 blocks are
> being moved from DN1 to DN2 in one iterations But MAX_NUM_CONCURRENT_MOVES in
> one iterations is hard coded to 5.
> Balancer report for 1st iteration:
> =================================
> {noformat}
> HOST-XX-XX-XX-XX:/home/Andreina/NewHadoop2nd/hadoop-2.0.0-SNAPSHOT/bin #
> ./hdfs balancer -threshold 1
> 12/05/03 17:31:28 INFO balancer.Balancer: Using a threshold of 1.0
> 12/05/03 17:31:28 INFO balancer.Balancer: namenodes =
> [hdfs://HOST-XX-XX-XX-XX:9002]
> 12/05/03 17:31:28 INFO balancer.Balancer: p =
> Balancer.Parameters[BalancingPolicy.Node, threshold=1.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> 12/05/03 17:31:30 INFO net.NetworkTopology: Adding a new node:
> /datacenter1/rack1/YY.YY.YY.YY:50176
> 12/05/03 17:31:30 INFO net.NetworkTopology: Adding a new node:
> /datacenter1/rack1/XX.XX.XX.XX:50076
> 12/05/03 17:31:30 INFO balancer.Balancer: 1 over-utilized:
> [Source[XX.XX.XX.XX:50076, utilization=5.018416429773605]]
> 12/05/03 17:31:30 INFO balancer.Balancer: 1 underutilized:
> [BalancerDatanode[YY.YY.YY.YY:50176, utilization=3.272819804269012E-5]]
> 12/05/03 17:31:30 INFO balancer.Balancer: Need to move 1.06 GB to make the
> cluster balanced.
> 12/05/03 17:31:30 INFO balancer.Balancer: Decided to move 716.13 MB bytes
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176
> 12/05/03 17:31:30 INFO balancer.Balancer: Will move 716.13 MB in this
> iteration
> May 3, 2012 5:31:30 PM 0 0 KB 1.06 GB
> 716.13 MB
> 12/05/03 17:35:29 INFO balancer.Balancer: Moving block -5275260117334749945
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:36:31 INFO balancer.Balancer: Moving block -8079758341763366944
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:37:12 INFO balancer.Balancer: Moving block -7395554712490186313
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:37:45 INFO balancer.Balancer: Moving block 7805443002654525130
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:38:15 INFO balancer.Balancer: Moving block 1864290085256894184
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:40:30 INFO balancer.Balancer: Moving block 23322655230037442 from
> XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is succeeded.
> 12/05/03 17:41:24 INFO balancer.Balancer: Moving block -8839566903692469634
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:43:03 INFO balancer.Balancer: Moving block 7304385435779271887
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:43:48 INFO balancer.Balancer: Moving block -7242009026552182303
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:44:06 INFO balancer.Balancer: Moving block -2449309138254106767
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:44:55 INFO balancer.Balancer: Moving block 500930296233438046
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.
> 12/05/03 17:45:04 INFO balancer.Balancer: Moving block 2642725820310610865
> from XX.XX.XX.XX:50076 to YY.YY.YY.YY:50176 through XX.XX.XX.XX:50076 is
> succeeded.{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
