[
https://issues.apache.org/jira/browse/HDFS-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802046#comment-15802046
]
Andrew Wang commented on HDFS-11285:
------------------------------------
Ah. So the issue is that there is a DN that still has a few blocks waiting for
decom (probably RBW blocks from an open file), and it transitions from (Live,
Decomming) to (Dead, Decomming), not (Dead, Decommed). We don't immediately
transition to (Dead, Decommed) since that could lead to unintentional dataloss.
If an operator wants to override this for a dead node, then they can run your
procedure.
Question, do these nodes truly *never* transition to (Dead, Decommed) ? Once
the straggler blocks are closed and minimally replicated, (Dead, Decomming)
should transition to (Dead, Decommed).
Otherwise, I'm guessing the straggler blocks are caused by open-for-write
files. Fixing that is more work; we could force the writing clients to do
pipeline recovery to route off the decomming nodes to maintain minimal
replication.
> Dead DataNodes keep a long time in (Dead, DECOMMISSION_INPROGRESS), and never
> transition to (Dead, DECOMMISSIONED)
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-11285
> URL: https://issues.apache.org/jira/browse/HDFS-11285
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.7.1
> Reporter: Lantao Jin
> Attachments: DecomStatus.png
>
>
> We have seen the use case of decommissioning DataNodes that are already dead
> or unresponsive, and not expected to rejoin the cluster. In a large cluster,
> we met more than 100 nodes were dead, decommissioning and their {panel} Under
> replicated blocks {panel} {panel} Blocks with no live replicas {panel} were
> all ZERO. Actually It has been fixed in
> [HDFS-7374|https://issues.apache.org/jira/browse/HDFS-7374]. After that, we
> can refreshNode twice to eliminate this case. But, seems this patch missed
> after refactor[HDFS-7411|https://issues.apache.org/jira/browse/HDFS-7411]. We
> are using a Hadoop version based 2.7.1 and only below operations can
> transition the status from {panel} Dead, DECOMMISSION_INPROGRESS {panel} to
> {panel} Dead, DECOMMISSIONED {panel}:
> # Retire it from hdfs-exclude
> # refreshNodes
> # Re-add it to hdfs-exclude
> # refreshNodes
> So, why the code removed after refactor in the new DecommissionManager?
> {code:java}
> if (!node.isAlive) {
> LOG.info("Dead node " + node + " is decommissioned immediately.");
> node.setDecommissioned();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
