[ https://issues.apache.org/jira/browse/HDFS-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865690#comment-16865690 ]
Wei-Chiu Chuang commented on HDFS-14576: ---------------------------------------- Additionally, [~hexiaoqiao] have you considered improving NN start time? For large clusters NN takes more than half hour just to load fsimage, even without block reports. I think there's room for improvement. For example, loading fsimage is a single thread operation. It shouldn't be too hard to parallelize it. > Avoid block report retry and slow down namenode startup > ------------------------------------------------------- > > Key: HDFS-14576 > URL: https://issues.apache.org/jira/browse/HDFS-14576 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode > Reporter: He Xiaoqiao > Assignee: He Xiaoqiao > Priority: Major > > During namenode startup, the load will be very high since it has to process > every datanodes blockreport one by one. If there are hundreds datanodes block > reports pending process, the issue will be more serious even > #processFirstBlockReport is processed a lot more efficiently than ordinary > block reports. Then some of datanode will retry blockreport and lengthens > restart times. I think we should filter the block report request (via > datanode blockreport retries) which has be processed and return directly then > shorten down restart time. I want to state this proposal may be obvious only > for large cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org