Hi, what is the location of the namenodes fsimage and editlogs? And how much memory has the NameNode.
Did you work with a Secondary NameNode or a Standby NameNode for checkpointing? Where are your HDFS blocks located, are those still safe? With this information at hand, one might be able to fix your setup, but do not format the old namenode before all is working with a fresh one. Grab a copy of the maintainance guide: http://shop.oreilly.com/product/0636920025085.do?sortby=publicationDate which helps solving such type of problems as well. Best wishes Mirko 2014-03-16 9:07 GMT+00:00 Fatih Haltas <fatih.hal...@nyu.edu>: > Dear All, > > I have just restarted machines of my hadoop clusters. Now, I am trying to > restart hadoop clusters again, but getting error on namenode restart. I am > afraid of loosing my data as it was properly running for more than 3 > months. Currently, I believe if I do namenode formatting, it will work > again, however, data will be lost. Is there anyway to solve this without > losing the data. > > I will really appreciate any help. > > Thanks. > > > ===================== > Here is the logs; > ==================== > 2014-02-26 16:02:39,698 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: > /************************************************************ > STARTUP_MSG: Starting NameNode > STARTUP_MSG: host = ADUAE042-LAP-V/127.0.0.1 > STARTUP_MSG: args = [] > STARTUP_MSG: version = 1.0.4 > STARTUP_MSG: build = > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r > 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012 > ************************************************************/ > 2014-02-26 16:02:40,005 INFO > org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from > hadoop-metrics2.properties > 2014-02-26 16:02:40,019 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > MetricsSystem,sub=Stats registered. > 2014-02-26 16:02:40,021 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > period at 10 second(s). > 2014-02-26 16:02:40,021 INFO > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system > started > 2014-02-26 16:02:40,169 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi > registered. > 2014-02-26 16:02:40,193 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm > registered. > 2014-02-26 16:02:40,194 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > NameNode registered. > 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: VM type > = 64-bit > 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: 2% max > memory = 17.77875 MB > 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: capacity > = 2^21 = 2097152 entries > 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: > recommended=2097152, actual=2097152 > 2014-02-26 16:02:40,273 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop > 2014-02-26 16:02:40,273 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup > 2014-02-26 16:02:40,274 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > isPermissionEnabled=true > 2014-02-26 16:02:40,279 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > dfs.block.invalidate.limit=100 > 2014-02-26 16:02:40,279 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: > isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), > accessTokenLifetime=0 min(s) > 2014-02-26 16:02:40,724 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered > FSNamesystemStateMBean and NameNodeMXBean > 2014-02-26 16:02:40,749 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names > occuring more than 10 times > 2014-02-26 16:02:40,780 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed. > java.io.IOException: NameNode is not formatted. > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288) > 2014-02-26 16:02:40,781 ERROR > org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: > NameNode is not formatted. > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288) > > 2014-02-26 16:02:40,781 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: > /************************************************************ > SHUTDOWN_MSG: Shutting down NameNode at ADUAE042-LAP-V/127.0.0.1 > ************************************************************/ > > =========================== > Here is the core-site.xml > =========================== > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > <!-- Put site-specific property overrides in this file. --> > > <configuration> > <property> > <name>fs.default.name</name> > <value>-BLANKED</value> > </property> > <property> > <name>hadoop.tmp.dir</name> > <value>/home/hadoop/project/hadoop-data</value> > </property> > </configuration> > > > >
