I think I know what the problem is... It was related to maximum number of file descriptors I can had opened. The limit was 1024 and I was exceeding that limit. In order to fix it I increased the number to 65535 using help from here http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html
The process takes 2 hrs to finish but I am hoping it does finish this time without causing any pains. Parag On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka <[email protected]>wrote: > I re ran after formatting and got this error.. Can it be because I have too > many files opened? > 2009-02-24 02:46:19,284 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 192.168.1.201:50010, storageID=DS-1261214527-127.0 > .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver > java.io.IOException: Input/output error > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.createNewFile(File.java:883) > at > org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398) > at > org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371) > at > org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108) > at > org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997) > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102) > at java.lang.Thread.run(Thread.java:619) > > On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka > <[email protected]>wrote: > >> Hi Mithila, >> Thanks for a quick reply... >> >> I did check the name node and data node logs... They have died and I ll >> restart them but I just want to know what happened before doing anything. >> >> On the data node the interesting error which I saw is this: >> >> >> java.io.EOFException: while trying to read 65557 bytes >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514) >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356) >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102) >> >> At the same time I saw the following error on name node >> 2009-02-23 10:02:57,181 >> FATAL org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : >> All storage directories are inaccessible. >> >> Can you please elaborate as to what this might be. >> >> Parag >> >> On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra <[email protected]>wrote: >> >>> Hey Parag >>> Check if the namenode and the datanode are up and running. Use the 'jps' >>> command to do so. If they are not running U ll have to do a stop-all and >>> reformat the namenode using hadoop namenode -format (make sure u have no >>> data on the HDFS). Then restart hadoop using start-all.sh. >>> >>> If U have the datanode and the namenode running, then check the log files >>> for errors. >>> >>> Mithila >>> >>> On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <[email protected] >>> >wrote: >>> >>> > I have setup hadoop in pseudo distributed mode with namenode, datanode, >>> > jobtracker and tasktracker all on the same machine... >>> > I also have a code which I use to write my data into hadoop. The code >>> of my >>> > mine reads data from the local disk does some preprocessing and after >>> that >>> > uses (multiple) FSDataOutputStream to write data to hadoop. I have >>> multiple >>> > FSDataOutputStreams open at one time cause I want to write data into >>> > different files based on some logic I have. >>> > >>> > Now the problem... While the process was writing data to hadoop I got >>> this >>> > error Problem renewing lease for DFSClient_1637324984. On going to name >>> > node >>> > logs I found this >>> > 2009-02-23 10:02:57,181 FATAL >>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All >>> > storage directories are inaccessible. >>> > >>> > I have absolutely no idea as to what might have caused this. Can some >>> one >>> > please help. >>> > >>> > -- >>> > Parag Dhanuka >>> > >>> >> >> >> >> -- >> Parag Dhanuka >> > > > > -- > Parag Dhanuka > -- Parag Dhanuka
