Can you also make sure your hostname and IP address are still mapped
correctly. Because what I am guessing is when you restart your machine,
your /etc/hosts entries might get restored (it happens in some
distributions, based on how you installed it). So when you are trying to
restart your namenode, it might be pointing to some different IP/machine
(in general localhost).
I can't think of any reason how it can happen just by restarting the
machine.
On Mon, Mar 24, 2014 at 5:42 AM, Stanley Shi <s...@gopivotal.com> wrote:
> Can you confirm that you namenode image and fseditlog are still there? if
> not, then your data IS lost.
>
> Regards,
> *Stanley Shi,*
>
>
>
> On Sun, Mar 23, 2014 at 6:24 PM, Fatih Haltas <fatih.hal...@nyu.edu>wrote:
>
>> No, not ofcourse I blinded it.
>>
>>
>> On Wed, Mar 19, 2014 at 5:09 PM, praveenesh kumar
>> <praveen...@gmail.com>wrote:
>>
>>> Is this property correct ?
>>>
>>>
>>> <property>
>>> <name>fs.default.name</name>
>>> <value>-BLANKED</value>
>>> </property>
>>>
>>> Regards
>>> Prav
>>>
>>>
>>> On Wed, Mar 19, 2014 at 12:58 PM, Fatih Haltas <fatih.hal...@nyu.edu>wrote:
>>>
>>>> Thanks for you helps, but still could not solve my problem.
>>>>
>>>>
>>>> On Tue, Mar 18, 2014 at 10:13 AM, Stanley Shi <s...@gopivotal.com>wrote:
>>>>
>>>>> Ah yes, I overlooked this. Then please check the file are there or
>>>>> not: "ls /home/hadoop/project/hadoop-data/dfs/name"?
>>>>>
>>>>> Regards,
>>>>> *Stanley Shi,*
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Mar 18, 2014 at 2:06 PM, Azuryy Yu <azury...@gmail.com> wrote:
>>>>>
>>>>>> I don't think this is the case, because there is;
>>>>>> <property>
>>>>>> <name>hadoop.tmp.dir</name>
>>>>>> <value>/home/hadoop/project/hadoop-data</value>
>>>>>> </property>
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 18, 2014 at 1:55 PM, Stanley Shi <s...@gopivotal.com>wrote:
>>>>>>
>>>>>>> one possible reason is that you didn't set the namenode working
>>>>>>> directory, by default it's in "/tmp" folder; and the "/tmp" folder might
>>>>>>> get deleted by the OS without any notification. If this is the case, I
>>>>>>> am
>>>>>>> afraid you have lost all your namenode data.
>>>>>>>
>>>>>>> *<property>
>>>>>>> <name>dfs.name.dir</name>
>>>>>>> <value>${hadoop.tmp.dir}/dfs/name</value>
>>>>>>> <description>Determines where on the local filesystem the DFS name
>>>>>>> node
>>>>>>> should store the name table(fsimage). If this is a
>>>>>>> comma-delimited list
>>>>>>> of directories then the name table is replicated in all of the
>>>>>>> directories, for redundancy. </description>
>>>>>>> </property>*
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> *Stanley Shi,*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 16, 2014 at 5:29 PM, Mirko Kämpf <mirko.kae...@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> what is the location of the namenodes fsimage and editlogs?
>>>>>>>> And how much memory has the NameNode.
>>>>>>>>
>>>>>>>> Did you work with a Secondary NameNode or a Standby NameNode for
>>>>>>>> checkpointing?
>>>>>>>>
>>>>>>>> Where are your HDFS blocks located, are those still safe?
>>>>>>>>
>>>>>>>> With this information at hand, one might be able to fix your setup,
>>>>>>>> but do not format the old namenode before
>>>>>>>> all is working with a fresh one.
>>>>>>>>
>>>>>>>> Grab a copy of the maintainance guide:
>>>>>>>> http://shop.oreilly.com/product/0636920025085.do?sortby=publicationDate
>>>>>>>> which helps solving such type of problems as well.
>>>>>>>>
>>>>>>>> Best wishes
>>>>>>>> Mirko
>>>>>>>>
>>>>>>>>
>>>>>>>> 2014-03-16 9:07 GMT+00:00 Fatih Haltas <fatih.hal...@nyu.edu>:
>>>>>>>>
>>>>>>>> Dear All,
>>>>>>>>>
>>>>>>>>> I have just restarted machines of my hadoop clusters. Now, I am
>>>>>>>>> trying to restart hadoop clusters again, but getting error on namenode
>>>>>>>>> restart. I am afraid of loosing my data as it was properly running
>>>>>>>>> for more
>>>>>>>>> than 3 months. Currently, I believe if I do namenode formatting, it
>>>>>>>>> will
>>>>>>>>> work again, however, data will be lost. Is there anyway to solve this
>>>>>>>>> without losing the data.
>>>>>>>>>
>>>>>>>>> I will really appreciate any help.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> =====================
>>>>>>>>> Here is the logs;
>>>>>>>>> ====================
>>>>>>>>> 2014-02-26 16:02:39,698 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
>>>>>>>>> /************************************************************
>>>>>>>>> STARTUP_MSG: Starting NameNode
>>>>>>>>> STARTUP_MSG: host = ADUAE042-LAP-V/127.0.0.1
>>>>>>>>> STARTUP_MSG: args = []
>>>>>>>>> STARTUP_MSG: version = 1.0.4
>>>>>>>>> STARTUP_MSG: build =
>>>>>>>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0-r
>>>>>>>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>>>>>>>> ************************************************************/
>>>>>>>>> 2014-02-26 16:02:40,005 INFO
>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>>>>>>>> hadoop-metrics2.properties
>>>>>>>>> 2014-02-26 16:02:40,019 INFO
>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>>>>>>> MetricsSystem,sub=Stats registered.
>>>>>>>>> 2014-02-26 16:02:40,021 INFO
>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>>>>>>>> period at 10 second(s).
>>>>>>>>> 2014-02-26 16:02:40,021 INFO
>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
>>>>>>>>> system
>>>>>>>>> started
>>>>>>>>> 2014-02-26 16:02:40,169 INFO
>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for
>>>>>>>>> source ugi
>>>>>>>>> registered.
>>>>>>>>> 2014-02-26 16:02:40,193 INFO
>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for
>>>>>>>>> source jvm
>>>>>>>>> registered.
>>>>>>>>> 2014-02-26 16:02:40,194 INFO
>>>>>>>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>>>>>>>> NameNode registered.
>>>>>>>>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: VM
>>>>>>>>> type = 64-bit
>>>>>>>>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet: 2%
>>>>>>>>> max memory = 17.77875 MB
>>>>>>>>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet:
>>>>>>>>> capacity = 2^21 = 2097152 entries
>>>>>>>>> 2014-02-26 16:02:40,242 INFO org.apache.hadoop.hdfs.util.GSet:
>>>>>>>>> recommended=2097152, actual=2097152
>>>>>>>>> 2014-02-26 16:02:40,273 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop
>>>>>>>>> 2014-02-26 16:02:40,273 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>>>>>> supergroup=supergroup
>>>>>>>>> 2014-02-26 16:02:40,274 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>>>>>> isPermissionEnabled=true
>>>>>>>>> 2014-02-26 16:02:40,279 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>>>>>> dfs.block.invalidate.limit=100
>>>>>>>>> 2014-02-26 16:02:40,279 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
>>>>>>>>> isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
>>>>>>>>> accessTokenLifetime=0 min(s)
>>>>>>>>> 2014-02-26 16:02:40,724 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
>>>>>>>>> FSNamesystemStateMBean and NameNodeMXBean
>>>>>>>>> 2014-02-26 16:02:40,749 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
>>>>>>>>> occuring more than 10 times
>>>>>>>>> 2014-02-26 16:02:40,780 ERROR
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
>>>>>>>>> initialization failed.
>>>>>>>>> java.io.IOException: NameNode is not formatted.
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>>>>>>>>> 2014-02-26 16:02:40,781 ERROR
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>>>>>>>>> NameNode is not formatted.
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:330)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:100)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:388)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:362)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>>>>>>>>>
>>>>>>>>> 2014-02-26 16:02:40,781 INFO
>>>>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>>>>>>>> /************************************************************
>>>>>>>>> SHUTDOWN_MSG: Shutting down NameNode at ADUAE042-LAP-V/127.0.0.1
>>>>>>>>> ************************************************************/
>>>>>>>>>
>>>>>>>>> ===========================
>>>>>>>>> Here is the core-site.xml
>>>>>>>>> ===========================
>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>>>>>
>>>>>>>>> <!-- Put site-specific property overrides in this file. -->
>>>>>>>>>
>>>>>>>>> <configuration>
>>>>>>>>> <property>
>>>>>>>>> <name>fs.default.name</name>
>>>>>>>>> <value>-BLANKED</value>
>>>>>>>>> </property>
>>>>>>>>> <property>
>>>>>>>>> <name>hadoop.tmp.dir</name>
>>>>>>>>> <value>/home/hadoop/project/hadoop-data</value>
>>>>>>>>> </property>
>>>>>>>>> </configuration>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
