[
https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553292#comment-14553292
]
Jing Zhao commented on HDFS-7609:
---------------------------------
Thanks for working on this, [~mingma]! Your analysis makes sense to me and I
can also reproduce the same scenario. Your patch also looks good to me.
One extra issue, which exists before your patch, is that the retry cache may
not be hit during the NameNode failover. For example, suppose the following
event sequence:
# a delete request is served by the ANN but the client fails to receive the
response
# NN failover happens
# the first retry request gets StandbyException since the old ANN becomes
standby
# the second retry request is sent to the other NN, which at this time has not
started the transition yet
# no cached entry has been found in the retry cache
# before running {{FSNamesystem#delete}}, the transition starts, which blocks
the {{FSNamesystem#delete}} call
# the {{FSNamesystem#delete}} is called and fails
If the above scenario stands, maybe we can use this chance to fix it? In your
patch, if the NN is in standby state, and there is no retry cache entry, should
we directly throw StandbyException instead of checking it again in
FSNamesystem?
> startup used too much time to load edits
> ----------------------------------------
>
> Key: HDFS-7609
> URL: https://issues.apache.org/jira/browse/HDFS-7609
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.2.0
> Reporter: Carrey Zhan
> Assignee: Ming Ma
> Labels: BB2015-05-RFC
> Attachments: HDFS-7609-CreateEditsLogWithRPCIDs.patch,
> HDFS-7609.patch, recovery_do_not_use_retrycache.patch
>
>
> One day my namenode crashed because of two journal node timed out at the same
> time under very high load, leaving behind about 100 million transactions in
> edits log.(I still have no idea why they were not rolled into fsimage.)
> I tryed to restart namenode, but it showed that almost 20 hours would be
> needed before finish, and it was loading fsedits most of the time. I also
> tryed to restart namenode in recover mode, the loading speed had no different.
> I looked into the stack trace, judged that it is caused by the retry cache.
> So I set dfs.namenode.enable.retrycache to false, the restart process
> finished in half an hour.
> I think the retry cached is useless during startup, at least during recover
> process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
