[
https://issues.apache.org/jira/browse/HDFS-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801657#comment-16801657
]
Wei-Chiu Chuang commented on HDFS-12049:
----------------------------------------
We saw this bug recently. As a data point, recommissioning a DataNode with ~2
million blocks stalled NN for nearly 5 minutes. Examine jstack collected during
the recomm, I suspect logging contributed partially to the slowdown. The
cluster's version doesn't have HDFS-6860, and so every invalidated block print
a line of INFO log.
HDFS-6860 should alleviate this pain, but in the end, it doesn't make much
sense to hold NN lock and iterate through all replicas on all recomm DNs. The
lock should be released periodically. HDFS-10477 reports a similar issue and a
patch is pending, but there are concurrency concerns with the patch if lock is
released while iterating block iterator.
> Recommissioning live nodes stalls the NN
> ----------------------------------------
>
> Key: HDFS-12049
> URL: https://issues.apache.org/jira/browse/HDFS-12049
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Daryn Sharp
> Priority: Critical
>
> A node refresh will recommission included nodes that are alive and in
> decommissioning or decommissioned state. The recommission will scan all
> blocks on the node, find over replicated blocks, chose an excess, queue an
> invalidate.
> The process is expensive and worsened by overhead of storage types (even when
> not in use). It can be especially devastating because the write lock is held
> for the entire node refresh. _Recommissioning 67 nodes with ~500k
> blocks/node stalled rpc services for over 4 mins._
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
