Hi, following is the output on the df command [r...@domu-12-31-39-00-e5-d2 conf]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 9.9G 4.2G 5.2G 45% / /dev/sdb 414G 924M 392G 1% /mnt
from the o/p it seems that i have quite an amount of memory available. but i still get the exception :( Thanks Raakhi On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B <milind.de...@hp.com>wrote: > From the exception it appears that there is no space left on machine. You > can check using 'df' > > Thanks > Milind > > -----Original Message----- > From: Rakhi Khatwani [mailto:rakhi.khatw...@gmail.com] > Sent: Thursday, April 16, 2009 1:15 PM > To: hbase-u...@hadoop.apache.org; core-user@hadoop.apache.org > Subject: No space left on device Exception > > Hi, > I am running a map-reduce program on 6-Node ec2 cluster. and after a > couple of hours all my tasks gets hanged. > > so i started digging into the logs.... > > there were no logs for regionserver > no logs for tasktracker. > However for jobtracker i get the following: > > 2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 9 on 50002, call > heartbeat(org.apache.hadoop.mapred.tasktrackersta...@2eed7d11, false, > true, > 10745) from 10.254.27.79:44222: error: java.io.IOException: > org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device > java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException: No > space left on device > at > > org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199) > at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at > > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at > > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346) > at > > org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150) > at > org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86) > at > > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47) > at java.io.DataOutputStream.write(DataOutputStream.java:90) > at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) > at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297) > at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130) > at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216) > at java.io.BufferedWriter.close(BufferedWriter.java:248) > at java.io.PrintWriter.close(PrintWriter.java:295) > at > > org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024) > at > org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906) > at org.apache.hadoop.mapred.JobInProgress.comp > > > > following are the disk information on dfs UI > domU-12-31-39-00-0C-A1< > http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F > >0In > Service413.380.8321.19391.360.2 > 94.672353 domU-12-31-39-00-16-F1< > http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F > >1In > Service413.380.4621.24391.670.11 > 94.752399 domU-12-31-39-00-45-71< > http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F > >1In > Service413.380.6421.34391.40.16 > 94.682303 domU-12-31-39-00-E5-D2< > http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F > >0In > Service413.380.6621.53391.180.16 > 94.632319 domU-12-31-39-01-64-12< > http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F > >2In > Service413.380.6421.24391.490.16 > 94.712264 domU-12-31-39-01-78-D1< > http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F > >0In > Service413.380.4921.24391.650.12 > 94.741952 > > I m using hadoop 0.19.0 and hbase 0.19.0 > > n googling the error i came arcoss the JIRA issue > http://issues.apache.org/jira/browse/HADOOP-4163 > > which says tht its been fixed in this version. :( > > Has anyone else come up with this exception? > > how do we check the maximum capacity for usable dfs and non usable dfs. > Thanks > Raakhi, >