[
https://issues.apache.org/jira/browse/HDFS-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981773#comment-13981773
]
Aaron T. Myers commented on HDFS-6289:
--------------------------------------
Hi Yongjun,
bq. When Standby NN receive the messages, it's actually aware of what message
it received, but as time goes by, the messages may become stale due to DN
reformatting. That's what you meant, right?
Nope, that's not what I was referring to. I was referring to why we queue
messages in the first place in the standby NN. This happens because the standby
NN is always a bit behind by the active in its knowledge of the namespace and
set of blocks which exist, so when the active NN allocates new blocks the DNs
will report things about these blocks to both NNs but the standby won't yet
know about the existence of the blocks those messages refer to. In some sense
these messages are from the future from the DN's perspective, so we queue them
up.
bq. Since the DN that the queued messages are associated with is reformatted,
these queued messages become stale and useless, and can be safely removed,
right? or my real question is, are there any messages that need to applied even
after DN reformatting?
No, if the storage ID of the DN is changing, we should assume that the DN no
longer has the blocks that these messages were previously referring to.
bq. I found a util function in DatanodeUtil.java...
OK, I'll make this change when I do the next rev of this patch.
bq. I think the testcase you wrote is very nice to demonstrate the problem.
Thanks!
> HA failover can fail if there are pending DN messages for DNs which no longer
> exist
> -----------------------------------------------------------------------------------
>
> Key: HDFS-6289
> URL: https://issues.apache.org/jira/browse/HDFS-6289
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 2.4.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Priority: Critical
> Attachments: HDFS-6289.patch
>
>
> In an HA setup, the standby NN may receive messages from DNs for blocks which
> the standby NN is not yet aware of. It queues up these messages and replays
> them when it next reads from the edit log or fails over. On a failover, all
> of these pending DN messages must be processed successfully in order for the
> failover to succeed. If one of these pending DN messages refers to a DN
> storageId that no longer exists (because the DN with that transfer address
> has been reformatted and has re-registered with the same transfer address)
> then on transition to active the NN will not be able to process this DN
> message and will suicide with an error like the following:
> {noformat}
> 2014-04-25 14:23:17,922 FATAL namenode.NameNode
> (NameNode.java:doImmediateShutdown(1525)) - Error encountered requiring NN
> shutdown. Shutting down immediately.
> java.io.IOException: Cannot mark
> blk_1073741825_900(stored=blk_1073741825_1001) as corrupt because datanode
> 127.0.0.1:33324 does not exist
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
