[
https://issues.apache.org/jira/browse/HDFS-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946944#comment-16946944
]
Stephen O'Donnell commented on HDFS-14854:
------------------------------------------
[~elgoiri] Thanks for the comments. I have had a quick look at what it would
take to move the existing monitor out of the DatanodeAdminManager class. It is
doable without too much effort, but it would cost a reasonable amount of change
to the implementation which is there and working, which makes this change more
difficult to review and risky.
Ultimately I would like to end up with 1 decommission monitor, however while I
believe the new implementation will work better than the existing, we won't
know until it is tried on real clusters. Therefore I would like to give the
option to use the new one, and if it does not work better, the existing is
there to fall back to. Another option may be to move the new monitor into a
standalone class with a different constructor, and leave the existing
implementation untouched. I will spend a bit of time looking at this option and
see what I come up with.
> Create improved decommission monitor implementation
> ---------------------------------------------------
>
> Key: HDFS-14854
> URL: https://issues.apache.org/jira/browse/HDFS-14854
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 3.3.0
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Attachments: Decommission_Monitor_V2_001.pdf, HDFS-14854.001.patch,
> HDFS-14854.002.patch, HDFS-14854.003.patch, HDFS-14854.004.patch
>
>
> In HDFS-13157, we discovered a series of problems with the current
> decommission monitor implementation, such as:
> * Blocks are replicated sequentially disk by disk and node by node, and
> hence the load is not spread well across the cluster
> * Adding a node for decommission can cause the namenode write lock to be
> held for a long time.
> * Decommissioning nodes floods the replication queue and under replicated
> blocks from a future node or disk failure may way for a long time before they
> are replicated.
> * Blocks pending replication are checked many times under a write lock
> before they are sufficiently replicate, wasting resources
> In this Jira I propose to create a new implementation of the decommission
> monitor that resolves these issues. As it will be difficult to prove one
> implementation is better than another, the new implementation can be enabled
> or disabled giving the option of the existing implementation or the new one.
> I will attach a pdf with some more details on the design and then a version 1
> patch shortly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
