[ https://issues.apache.org/jira/browse/HDFS-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243424#comment-14243424 ]
Chris Nauroth commented on HDFS-7470: ------------------------------------- Creating a new {{FSNamesystem}} instance without running the full shutdown sequence on the old one would create a risk of some dangerous side effects. * A new namesystem lock instance would get created, and there would be no synchronization of multiple threads around this. This could violate mutual exclusion. Two different threads could hold the 2 different lock instances, and think that mutual exclusion has been enforced. * We wouldn't reap background threads inside things like the {{BlockManager}} and {{CacheManager}}. Over time, we'd slowly leak threads and eventually hit {{OutOfMemoryError}} conditions. * I can't remember if we hold an open file descriptor on the edit log when running as SecondaryNameNode. If we do, then discarding the old {{FSNamesystem}} without a proper shutdown would leak a file descriptor. In general, there are widespread assumptions throughout the codebase that {{FSNamesystem}} is instantiated exactly once and retained for the entire process lifetime. > SecondaryNameNode need twice memory when calling reloadFromImageFile > -------------------------------------------------------------------- > > Key: HDFS-7470 > URL: https://issues.apache.org/jira/browse/HDFS-7470 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: zhaoyunjiong > Assignee: zhaoyunjiong > Attachments: HDFS-7470.1.patch, HDFS-7470.patch > > > histo information at 2014-12-02 01:19 > {quote} > num #instances #bytes class name > ---------------------------------------------- > 1: 186449630 19326123016 [Ljava.lang.Object; > 2: 157366649 15107198304 > org.apache.hadoop.hdfs.server.namenode.INodeFile > 3: 183409030 11738177920 > org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo > 4: 157358401 5244264024 > [Lorg.apache.hadoop.hdfs.server.blockmanagement.BlockInfo; > 5: 3 3489661000 > [Lorg.apache.hadoop.util.LightWeightGSet$LinkedElement; > 6: 29253275 1872719664 [B > 7: 3230821 284312248 > org.apache.hadoop.hdfs.server.namenode.INodeDirectory > 8: 2756284 110251360 java.util.ArrayList > 9: 469158 22519584 org.apache.hadoop.fs.permission.AclEntry > 10: 847 17133032 [Ljava.util.HashMap$Entry; > 11: 188471 17059632 [C > 12: 314614 10067656 > [Lorg.apache.hadoop.hdfs.server.namenode.INode$Feature; > 13: 234579 9383160 > com.google.common.collect.RegularImmutableList > 14: 49584 6850280 <constMethodKlass> > 15: 49584 6356704 <methodKlass> > 16: 187270 5992640 java.lang.String > 17: 234579 5629896 > org.apache.hadoop.hdfs.server.namenode.AclFeature > {quote} > histo information at 2014-12-02 01:32 > {quote} > num #instances #bytes class name > ---------------------------------------------- > 1: 355838051 35566651032 [Ljava.lang.Object; > 2: 302272758 29018184768 > org.apache.hadoop.hdfs.server.namenode.INodeFile > 3: 352500723 22560046272 > org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo > 4: 302264510 10075087952 > [Lorg.apache.hadoop.hdfs.server.blockmanagement.BlockInfo; > 5: 177120233 9374983920 [B > 6: 3 3489661000 > [Lorg.apache.hadoop.util.LightWeightGSet$LinkedElement; > 7: 6191688 544868544 > org.apache.hadoop.hdfs.server.namenode.INodeDirectory > 8: 2799256 111970240 java.util.ArrayList > 9: 890728 42754944 org.apache.hadoop.fs.permission.AclEntry > 10: 330986 29974408 [C > 11: 596871 19099880 > [Lorg.apache.hadoop.hdfs.server.namenode.INode$Feature; > 12: 445364 17814560 > com.google.common.collect.RegularImmutableList > 13: 844 17132816 [Ljava.util.HashMap$Entry; > 14: 445364 10688736 > org.apache.hadoop.hdfs.server.namenode.AclFeature > 15: 329789 10553248 java.lang.String > 16: 91741 8807136 > org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction > 17: 49584 6850280 <constMethodKlass> > {quote} > And the stack trace shows it was doing reloadFromImageFile: > {quote} > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getInode(FSDirectory.java:2426) > at > org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectorySection(FSImageFormatPBINode.java:160) > at > org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243) > at > org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168) > at > org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:121) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:902) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:888) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.reloadFromImageFile(FSImage.java:562) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:1048) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:536) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:388) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:354) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:356) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1630) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:413) > at > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:350) > at java.lang.Thread.run(Thread.java:745) > {quote} > So before doing reloadFromImageFile, I think we need release old namesystem > to prevent SecondaryNameNode OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)