[
https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741318#comment-16741318
]
He Xiaoqiao commented on HDFS-14186:
------------------------------------
Hi [~elgoiri],
{quote}I'm guessing that the lifeline doesn't help because the DN is not
registered at all?{quote}
Yep, you are correct, lifeline do not help absolutely. Actually I have open the
feature in my production cluster.
{quote}I'm also curious about the numbers, 300M blocks takes 8 hours? We see
around 30 minutes with 60M block.{quote}
I found the case to different degrees with above 300M blocks. The worst case
which takes more than 8 hours I met: more than 15K nodes with more than 500M
blocks.
Generally, I think number of slave nodes and block numbers both can impact the
startup times under the same fsimage+editlogs. blocks number is more affected
factor of course. 30min can finish to restart with 60M blocks, I see the same
result as [~elgoiri] mentioned. Actually, with 100M blocks or less we can
control startup time within ~40min. It appears with more large-scale. more
comments are welcome.
> blockreport storm slow down namenode restart seriously in large cluster
> -----------------------------------------------------------------------
>
> Key: HDFS-14186
> URL: https://issues.apache.org/jira/browse/HDFS-14186
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: He Xiaoqiao
> Assignee: He Xiaoqiao
> Priority: Major
> Attachments: HDFS-14186.001.patch
>
>
> In the current implementation, the datanode sends blockreport immediately
> after register to namenode successfully when restart, and the blockreport
> storm will make namenode high load to process them. One result is some
> received RPC have to skip because queue time is timeout. If some datanodes'
> heartbeat RPC are continually skipped for long times (default is
> heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to
> re-register and send blockreport again, aggravate blockreport storm and trap
> in a vicious circle, and slow down (more than one hour and even more)
> namenode startup seriously in a large (several thousands of datanodes) and
> busy cluster especially. Although there are many work to optimize namenode
> startup, the issue still exists.
> I propose to postpone dead datanode check when namenode have finished startup.
> Any comments and suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
