[ https://issues.apache.org/jira/browse/HDFS-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566193#comment-13566193 ]
Chris Nauroth commented on HDFS-4423: ------------------------------------- Thank you, Nicholas! From my own test run, the only failure I saw was {{TestJvmReuse}}. This test has been failing consistently for me on branch-1, even before this patch. I will file a separate jira for follow-up on that. > Checkpoint exception causes fatal damage to fsimage. > ---------------------------------------------------- > > Key: HDFS-4423 > URL: https://issues.apache.org/jira/browse/HDFS-4423 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode > Affects Versions: 1.0.4, 1.1.1 > Environment: CentOS 6.2 > Reporter: ChenFolin > Assignee: Chris Nauroth > Priority: Blocker > Fix For: 1.1.2 > > Attachments: HDFS-4423-branch-1.1.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > The impact of class is org.apache.hadoop.hdfs.server.namenode.FSImage.java > {code} > boolean loadFSImage(MetaRecoveryContext recovery) throws IOException { > ... > latestNameSD.read(); > needToSave |= loadFSImage(getImageFile(latestNameSD, NameNodeFile.IMAGE)); > LOG.info("Image file of size " + imageSize + " loaded in " > + (FSNamesystem.now() - startTime)/1000 + " seconds."); > > // Load latest edits > if (latestNameCheckpointTime > latestEditsCheckpointTime) > // the image is already current, discard edits > needToSave |= true; > else // latestNameCheckpointTime == latestEditsCheckpointTime > needToSave |= (loadFSEdits(latestEditsSD, recovery) > 0); > > return needToSave; > } > {code} > If it is the normal flow of the checkpoint,the value of > latestNameCheckpointTime is equal to the value of > latestEditsCheckpointTime,and it will exec “else”. > The problem is that,latestNameCheckpointTime > latestEditsCheckpointTime: > SecondNameNode starts checkpoint, > ... > NameNode:rollFSImage,NameNode shutdown after write latestNameCheckpointTime > and before write latestEditsCheckpointTime. > Start NameNode:because latestNameCheckpointTime > > latestEditsCheckpointTime,so the value of needToSave is true, and it will not > update “rootDir”'s nsCount that is the cluster's file number(update exec at > loadFSEdits > “FSNamesystem.getFSNamesystem().dir.updateCountForINodeWithQuota()”),and then > “saveNamespace” will write file number to fsimage whit default value “1”。 > The next time,loadFSImage will fail. > Maybe,it will work: > {code} > boolean loadFSImage(MetaRecoveryContext recovery) throws IOException { > ... > latestNameSD.read(); > needToSave |= loadFSImage(getImageFile(latestNameSD, NameNodeFile.IMAGE)); > LOG.info("Image file of size " + imageSize + " loaded in " > + (FSNamesystem.now() - startTime)/1000 + " seconds."); > > // Load latest edits > if (latestNameCheckpointTime > latestEditsCheckpointTime){ > // the image is already current, discard edits > needToSave |= true; > FSNamesystem.getFSNamesystem().dir.updateCountForINodeWithQuota(); > } > else // latestNameCheckpointTime == latestEditsCheckpointTime > needToSave |= (loadFSEdits(latestEditsSD, recovery) > 0); > > return needToSave; > } > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira