[
https://issues.apache.org/jira/browse/HDFS-11742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993870#comment-15993870
]
Tsz Wo Nicholas Sze commented on HDFS-11742:
--------------------------------------------
> ... The reason is explained in the jira comments. ...
I think you mean the following comment:
{quote}
..., but the design is still flawed. Move decision is made then thread pools
are created, which likely only cover a subset of targets in the schedule. Also,
the schedule will less likely make use of all 50 threads anyway. In any case,
the moves will all pile up to the first 25 targets (in our case, 25 thread
pools seemed to be the limit). I don't think it will be faster than the
previous one.
{quote}
If 50 is not a good value in your case, you may change it by setting
dfs.datanode.balance.max.concurrent.moves.
> Revert the core changes from HDFS-8818
> --------------------------------------
>
> Key: HDFS-11742
> URL: https://issues.apache.org/jira/browse/HDFS-11742
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Priority: Blocker
> Attachments: HDFS-11742.branch-2.8.patch, HDFS-11742.branch-2.patch,
> HDFS-11742.trunk.patch
>
>
> This is to revert the core changes made by HDFS-8818. The reason is explained
> in the jira comments. HDFS-8818 put in config and logging changes that are
> tied to the core change. I will leave them as is.
> We ran 2.8 balancer with HDFS-8818 on a 280-node and a 2,400-node cluster. In
> both cases, it would hang forever after two iterations. The two iterations
> were also moving things at a significantly lower rate. The hang itself is
> fixed by HDFS-11377, but the design limitation remains, so the balancer
> throughput ends up actually lower.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
