[
https://issues.apache.org/jira/browse/HDFS-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981757#comment-13981757
]
Yongjun Zhang commented on HDFS-6289:
-------------------------------------
Hi ATM,
Thanks a lot for discovering this tricky problem and providing the fix. As a
learning experience, I read through the patch and had a few comments actually
mainly questions):
1. "In an HA setup, the standby NN may receive messages from DNs for blocks
which the standby NN is not yet aware of." When Standby NN receive the
messages, it's actually aware of what message it received, but as time goes by,
the messages may become stale due to DN reformatting. That's what you meant,
right?
2. Since the DN that the queued messages are associated with is reformatted,
these queued messages become stale and useless, and can be safely removed,
right? or my real question is, are there any messages that need to applied even
after DN reformatting?
3. I found a util function in DatanodeUtil.java
{code}
public static String getMetaName(String blockName, long generationStamp) {
return blockName + "_" + generationStamp + Block.METADATA_EXTENSION;
}
{code}
that you might consider using in
{code}
public static boolean changeGenStampOfBlock(int dnIndex, ExtendedBlock blk,
long newGenStamp) throws IOException {
{code}
4. I think the testcase you wrote is very nice to demonstrate the problem.
Thanks.
> HA failover can fail if there are pending DN messages for DNs which no longer
> exist
> -----------------------------------------------------------------------------------
>
> Key: HDFS-6289
> URL: https://issues.apache.org/jira/browse/HDFS-6289
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 2.4.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Priority: Critical
> Attachments: HDFS-6289.patch
>
>
> In an HA setup, the standby NN may receive messages from DNs for blocks which
> the standby NN is not yet aware of. It queues up these messages and replays
> them when it next reads from the edit log or fails over. On a failover, all
> of these pending DN messages must be processed successfully in order for the
> failover to succeed. If one of these pending DN messages refers to a DN
> storageId that no longer exists (because the DN with that transfer address
> has been reformatted and has re-registered with the same transfer address)
> then on transition to active the NN will not be able to process this DN
> message and will suicide with an error like the following:
> {noformat}
> 2014-04-25 14:23:17,922 FATAL namenode.NameNode
> (NameNode.java:doImmediateShutdown(1525)) - Error encountered requiring NN
> shutdown. Shutting down immediately.
> java.io.IOException: Cannot mark
> blk_1073741825_900(stored=blk_1073741825_1001) as corrupt because datanode
> 127.0.0.1:33324 does not exist
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
