one possible reason is that you didn't set the namenode working directory,
by default it's in "/tmp" folder; and the "/tmp" folder might get deleted
by the OS without any notification. If this is the case, I am afraid you
have lost all your namenode data.
*<property>
<name>dfs.name.dir</name>
<value>${hadoop.tmp.dir}/dfs/name</value>
<description>Determines where on the local filesystem the DFS name node
should store the name table(fsimage). If this is a comma-delimited list
of directories then the name table is replicated in all of the
directories, for redundancy. </description>
</property>*
Regards,
*Stanley Shi,*
On Sun, Mar 16, 2014 at 5:29 PM, Mirko Kämpf <mirko.kae...@gmail.com> wrote:
> Hi,
>
> what is the location of the namenodes fsimage and editlogs?
> And how much memory has the NameNode.
>
> Did you work with a Secondary NameNode or a Standby NameNode for
> checkpointing?
>
> Where are your HDFS blocks located, are those still safe?
>
> With this information at hand, one might be able to fix your setup, but do
> not format the old namenode before
> all is working with a fresh one.
>
> Grab a copy of the maintainance guide:
> http://shop.oreilly.com/product/0636920025085.do?sortby=publicationDate
> which helps solving such type of problems as well.
>
> Best wishes
> Mirko
>
>
> 2014-03-16 9:07 GMT+00:00 Fatih Haltas <fatih.hal...@nyu.edu>:
>
> Dear All,
>>
>> I have just restarted machines of my hadoop clusters. Now, I am trying to
>> restart hadoop clusters again, but getting error on namenode restart. I am
>> afraid of loosing my data as it was properly running for more than 3
>> months. Currently, I believe if I do namenode formatting, it will work
>> again, however, data will be lost. Is there anyway to solve this without
>> losing the data.
>>
>> I will really appreciate any help.
>>
>> Thanks.
>>
>>
>> =====================
>> Here is the logs;
>> ====================
>> 2014-02-26 16:02:39,698 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting NameNode
>> STARTUP_MSG: host = ADUAE042-LAP-V/127.0.0.1
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2014-02-26 16:02:40,005 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2014-02-26 16:02:40,019 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2014-02-26 16:02:40,021 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2014-02-26 16:02:40,021 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
>> started
>> 2014-02-26 16:02:40,169 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2014-02-26 16:02:40,193 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
>> registered.
>> 2014-02-26 16:02:40,194 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> NameNode registered.
>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: VM type
>> = 64-bit
>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: 2% max
>> memory = 17.77875 MB
>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: capacity
>> = 2^21 = 2097152 entries
>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet:
>> recommended=2097152, actual=2097152
>> 2014-02-26 16:02:40,273 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop
>> 2014-02-26 16:02:40,273 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
>> 2014-02-26 16:02:40,274 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isPermissionEnabled=true
>> 2014-02-26 16:02:40,279 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> dfs.block.invalidate.limit=100
>> 2014-02-26 16:02:40,279 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>> isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
>> accessTokenLifetime=0 min(s)
>> 2014-02-26 16:02:40,724 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
>> FSNamesystemStateMBean and NameNodeMXBean
>> 2014-02-26 16:02:40,749 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>> occuring more than 10 times
>> 2014-02-26 16:02:40,780 ERROR
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
>> initialization failed.
>> java.io.IOException: NameNode is not formatted.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>> 2014-02-26 16:02:40,781 ERROR
>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>> NameNode is not formatted.
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>>
>> 2014-02-26 16:02:40,781 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down NameNode at ADUAE042-LAP-V/127.0.0.1
>> ************************************************************/
>>
>> ===========================
>> Here is the core-site.xml
>> ===========================
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>> <!-- Put site-specific property overrides in this file. -->
>>
>> <configuration>
>> <property>
>> <name>fs.default.name</name>
>> <value>-BLANKED</value>
>> </property>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/home/hadoop/project/hadoop-data</value>
>> </property>
>> </configuration>
>>
>>
>>
>>
>
