[
https://issues.apache.org/jira/browse/HDFS-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566193#comment-13566193
]
Chris Nauroth commented on HDFS-4423:
-------------------------------------
Thank you, Nicholas! From my own test run, the only failure I saw was
{{TestJvmReuse}}. This test has been failing consistently for me on branch-1,
even before this patch. I will file a separate jira for follow-up on that.
> Checkpoint exception causes fatal damage to fsimage.
> ----------------------------------------------------
>
> Key: HDFS-4423
> URL: https://issues.apache.org/jira/browse/HDFS-4423
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 1.0.4, 1.1.1
> Environment: CentOS 6.2
> Reporter: ChenFolin
> Assignee: Chris Nauroth
> Priority: Blocker
> Fix For: 1.1.2
>
> Attachments: HDFS-4423-branch-1.1.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> The impact of class is org.apache.hadoop.hdfs.server.namenode.FSImage.java
> {code}
> boolean loadFSImage(MetaRecoveryContext recovery) throws IOException {
> ...
> latestNameSD.read();
> needToSave |= loadFSImage(getImageFile(latestNameSD, NameNodeFile.IMAGE));
> LOG.info("Image file of size " + imageSize + " loaded in "
> + (FSNamesystem.now() - startTime)/1000 + " seconds.");
>
> // Load latest edits
> if (latestNameCheckpointTime > latestEditsCheckpointTime)
> // the image is already current, discard edits
> needToSave |= true;
> else // latestNameCheckpointTime == latestEditsCheckpointTime
> needToSave |= (loadFSEdits(latestEditsSD, recovery) > 0);
>
> return needToSave;
> }
> {code}
> If it is the normal flow of the checkpoint,the value of
> latestNameCheckpointTime is equal to the value of
> latestEditsCheckpointTime，and it will exec “else”.
> The problem is that，latestNameCheckpointTime > latestEditsCheckpointTime：
> SecondNameNode starts checkpoint，
> ...
> NameNode：rollFSImage，NameNode shutdown after write latestNameCheckpointTime
> and before write latestEditsCheckpointTime.
> Start NameNode：because latestNameCheckpointTime >
> latestEditsCheckpointTime，so the value of needToSave is true， and it will not
> update “rootDir”'s nsCount that is the cluster's file number（update exec at
> loadFSEdits
> “FSNamesystem.getFSNamesystem().dir.updateCountForINodeWithQuota()”），and then
> “saveNamespace” will write file number to fsimage whit default value “1”。
> The next time，loadFSImage will fail.
> Maybe，it will work:
> {code}
> boolean loadFSImage(MetaRecoveryContext recovery) throws IOException {
> ...
> latestNameSD.read();
> needToSave |= loadFSImage(getImageFile(latestNameSD, NameNodeFile.IMAGE));
> LOG.info("Image file of size " + imageSize + " loaded in "
> + (FSNamesystem.now() - startTime)/1000 + " seconds.");
>
> // Load latest edits
> if (latestNameCheckpointTime > latestEditsCheckpointTime){
> // the image is already current, discard edits
> needToSave |= true;
> FSNamesystem.getFSNamesystem().dir.updateCountForINodeWithQuota();
> }
> else // latestNameCheckpointTime == latestEditsCheckpointTime
> needToSave |= (loadFSEdits(latestEditsSD, recovery) > 0);
>
> return needToSave;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
