[
https://issues.apache.org/jira/browse/HDFS-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17043367#comment-17043367
]
Stephen O'Donnell commented on HDFS-14854:
------------------------------------------
Hi [~ayushtkn] there are quite a few factors to consider in this.
Firstly, decommission speed is mostly dictated by the speed the replication
manager works at, and nothing about that has changed with this patch.
One thing we did attempt to do, was ensure that the blocks were shuffled so
that if the decommissioning node has many disks, it should not pick blocks only
from the same disk, which is what the origional monitor did. Depending on your
settings for max-streams and work-multiplier, there may not be enough blocks
moving at the same time to saturate a disk and therefore you would not see any
benefit of this.
This change should also result in less load on the NN. It scans the blocks less
often to check if they have completed replication, does a lot of work under the
read lock rather than write lock. For maintenance, it should also result in
nodes going into maintenance faster provided they don't need to do any
replication.
Even if replication happens at the same speed, the new monitor should use less
CPU and cause less lock contention on the NN, which is a good thing, but very
hard to measure.
> Create improved decommission monitor implementation
> ---------------------------------------------------
>
> Key: HDFS-14854
> URL: https://issues.apache.org/jira/browse/HDFS-14854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 3.3.0
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Fix For: 3.3.0
>
> Attachments: 012_to_013_changes.diff,
> Decommission_Monitor_V2_001.pdf, HDFS-14854.001.patch, HDFS-14854.002.patch,
> HDFS-14854.003.patch, HDFS-14854.004.patch, HDFS-14854.005.patch,
> HDFS-14854.006.patch, HDFS-14854.007.patch, HDFS-14854.008.patch,
> HDFS-14854.009.patch, HDFS-14854.010.patch, HDFS-14854.011.patch,
> HDFS-14854.012.patch, HDFS-14854.013.patch, HDFS-14854.014.patch
>
>
> In HDFS-13157, we discovered a series of problems with the current
> decommission monitor implementation, such as:
> * Blocks are replicated sequentially disk by disk and node by node, and
> hence the load is not spread well across the cluster
> * Adding a node for decommission can cause the namenode write lock to be
> held for a long time.
> * Decommissioning nodes floods the replication queue and under replicated
> blocks from a future node or disk failure may way for a long time before they
> are replicated.
> * Blocks pending replication are checked many times under a write lock
> before they are sufficiently replicate, wasting resources
> In this Jira I propose to create a new implementation of the decommission
> monitor that resolves these issues. As it will be difficult to prove one
> implementation is better than another, the new implementation can be enabled
> or disabled giving the option of the existing implementation or the new one.
> I will attach a pdf with some more details on the design and then a version 1
> patch shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
