Sorry for spamming but I also found some interesting things in the secondary data node and here is the log again. My name node crashed at 2009-02-24 06:02:38,771 FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All storage directories are inaccessible.
whilst secondary data node had this 2009-02-24 05:02:38,741 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done. New Image Size: 28519 2009-02-24 06:02:38,757 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0 Number of syncs: 0 SyncTimes(ms): 0 2009-02-24 06:02:39,089 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint: 2009-02-24 06:02:39,089 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.IOException: Call to hadoop1/192.168.1.201:9000 failed : Can checkpointing be creating some problem.. Parag On Tue, Feb 24, 2009 at 7:46 PM, Parag Dhanuka <[email protected]>wrote: > This is how my data log looks like..... Please help.... I am lost now :( > 2009-02-24 05:52:17,613 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block > blk_-7039044529189438628_1012 terminating > 2009-02-24 05:52:17,619 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / > 192.168.1.200:55700, dest: /192.168.1.201:50010, byte > : 22260762, op: HDFS_WRITE, cliID: DFSClient_1664136935, srvID: > DS-1255721471-127.0.1.1-50010-1235480246191, blockid: > blk_7602190212483678862_1017 > 2009-02-24 05:52:17,619 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block > blk_7602190212483678862_1017 terminating > 2009-02-24 05:55:54,536 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_-477808440580664947_1021 > 2009-02-24 05:56:54,539 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_1534358347758148859_1021 > 2009-02-24 06:01:38,546 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification > succeeded for blk_-3560215495928841638_1004 > 2009-02-24 06:02:41,560 WARN > org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call > to hadoop1/192.168.1.201:9000 failed on local > exception: null > at org.apache.hadoop.ipc.Client.call(Client.java:699) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) > at $Proxy4.sendHeartbeat(Unknown Source) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:669) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1100) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:493) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:438) > > Parag > > On Tue, Feb 24, 2009 at 7:43 PM, Parag Dhanuka <[email protected]>wrote: > >> It doesnot seem to be helping.... my name node got down again and on >> trying to restart it I got this error.... >> 2009-02-24 06:06:59,063 ERROR >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem >> initialization failed. >> java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock >> (Input/output error) >> at java.io.RandomAccessFile.open(Native Method) >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) >> at >> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524) >> at >> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505) >> at >> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273) >> at >> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868) >> 2009-02-24 06:06:59,065 INFO org.apache.hadoop.ipc.Server: Stopping server >> on 9000 >> 2009-02-24 06:06:59,065 ERROR >> org.apache.hadoop.hdfs.server.namenode.NameNode: >> java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock (Inpu >> /output error) >> at java.io.RandomAccessFile.open(Native Method) >> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212) >> at >> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524) >> at >> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505) >> at >> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363) >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273) >> at >> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868) >> >> >> >> I am lost please helpppppppppppp >> >> Parag >> >> On Tue, Feb 24, 2009 at 7:01 PM, Parag Dhanuka >> <[email protected]>wrote: >> >>> I think I know what the problem is... It was related to maximum number of >>> file descriptors I can had opened. The limit was 1024 and I was exceeding >>> that limit. In order to fix it I increased the number to 65535 using help >>> from here >>> http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html >>> >>> The process takes 2 hrs to finish but I am hoping it does finish this >>> time without causing any pains. >>> >>> Parag >>> >>> >>> On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka >>> <[email protected]>wrote: >>> >>>> I re ran after formatting and got this error.. Can it be because I have >>>> too many files opened? >>>> 2009-02-24 02:46:19,284 ERROR >>>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( >>>> 192.168.1.201:50010, storageID=DS-1261214527-127.0 >>>> .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver >>>> java.io.IOException: Input/output error >>>> at java.io.UnixFileSystem.createFileExclusively(Native Method) >>>> at java.io.File.createNewFile(File.java:883) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258) >>>> at >>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102) >>>> at java.lang.Thread.run(Thread.java:619) >>>> >>>> On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka < >>>> [email protected]> wrote: >>>> >>>>> Hi Mithila, >>>>> Thanks for a quick reply... >>>>> >>>>> I did check the name node and data node logs... They have died and I ll >>>>> restart them but I just want to know what happened before doing anything. >>>>> >>>>> On the data node the interesting error which I saw is this: >>>>> >>>>> >>>>> java.io.EOFException: while trying to read 65557 bytes >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356) >>>>> at >>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102) >>>>> >>>>> At the same time I saw the following error on name node >>>>> 2009-02-23 10:02:57,181 >>>>> FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : >>>>> All storage directories are inaccessible. >>>>> >>>>> Can you please elaborate as to what this might be. >>>>> >>>>> Parag >>>>> >>>>> On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra >>>>> <[email protected]>wrote: >>>>> >>>>>> Hey Parag >>>>>> Check if the namenode and the datanode are up and running. Use the >>>>>> 'jps' >>>>>> command to do so. If they are not running U ll have to do a stop-all >>>>>> and >>>>>> reformat the namenode using hadoop namenode -format (make sure u have >>>>>> no >>>>>> data on the HDFS). Then restart hadoop using start-all.sh. >>>>>> >>>>>> If U have the datanode and the namenode running, then check the log >>>>>> files >>>>>> for errors. >>>>>> >>>>>> Mithila >>>>>> >>>>>> On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka < >>>>>> [email protected]>wrote: >>>>>> >>>>>> > I have setup hadoop in pseudo distributed mode with namenode, >>>>>> datanode, >>>>>> > jobtracker and tasktracker all on the same machine... >>>>>> > I also have a code which I use to write my data into hadoop. The >>>>>> code of my >>>>>> > mine reads data from the local disk does some preprocessing and >>>>>> after that >>>>>> > uses (multiple) FSDataOutputStream to write data to hadoop. I have >>>>>> multiple >>>>>> > FSDataOutputStreams open at one time cause I want to write data into >>>>>> > different files based on some logic I have. >>>>>> > >>>>>> > Now the problem... While the process was writing data to hadoop I >>>>>> got this >>>>>> > error Problem renewing lease for DFSClient_1637324984. On going to >>>>>> name >>>>>> > node >>>>>> > logs I found this >>>>>> > 2009-02-23 10:02:57,181 FATAL >>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : >>>>>> All >>>>>> > storage directories are inaccessible. >>>>>> > >>>>>> > I have absolutely no idea as to what might have caused this. Can >>>>>> some one >>>>>> > please help. >>>>>> > >>>>>> > -- >>>>>> > Parag Dhanuka >>>>>> > >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Parag Dhanuka >>>>> >>>> >>>> >>>> >>>> -- >>>> Parag Dhanuka >>>> >>> >>> >>> >>> -- >>> Parag Dhanuka >>> >> >> >> >> -- >> Parag Dhanuka >> > > > > -- > Parag Dhanuka > -- Parag Dhanuka
