Thanks,
I will check tht
Regards,
Raakhi
On Thu, Apr 16, 2009 at 1:42 PM, Miles Osborne <mi...@inf.ed.ac.uk> wrote:
> it may be that intermediate results are filling your disks and when
> the jobs crash, this all gets deleted. so it would look like you have
> spare space when in reality you don't.
>
> i would check on the file system as your jobs run and see if indeed
> they are filling-up.
>
> Miles
>
> 2009/4/16 Rakhi Khatwani <rakhi.khatw...@gmail.com>:
> > Hi,
> > following is the output on the df command
> > [r...@domu-12-31-39-00-e5-d2 conf]# df -h
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/sda1 9.9G 4.2G 5.2G 45% /
> > /dev/sdb 414G 924M 392G 1% /mnt
> >
> > from the o/p it seems that i have quite an amount of memory available.
> but i
> > still get the exception :(
> >
> > Thanks
> > Raakhi
> >
> > On Thu, Apr 16, 2009 at 1:18 PM, Desai, Milind B <milind.de...@hp.com
> >wrote:
> >
> >> From the exception it appears that there is no space left on machine.
> You
> >> can check using 'df'
> >>
> >> Thanks
> >> Milind
> >>
> >> -----Original Message-----
> >> From: Rakhi Khatwani [mailto:rakhi.khatw...@gmail.com]
> >> Sent: Thursday, April 16, 2009 1:15 PM
> >> To: hbase-u...@hadoop.apache.org; core-user@hadoop.apache.org
> >> Subject: No space left on device Exception
> >>
> >> Hi,
> >> I am running a map-reduce program on 6-Node ec2 cluster. and after a
> >> couple of hours all my tasks gets hanged.
> >>
> >> so i started digging into the logs....
> >>
> >> there were no logs for regionserver
> >> no logs for tasktracker.
> >> However for jobtracker i get the following:
> >>
> >> 2009-04-16 03:00:29,691 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> handler 9 on 50002, call
> >> heartbeat(org.apache.hadoop.mapred.tasktrackersta...@2eed7d11, false,
> >> true,
> >> 10745) from 10.254.27.79:44222: error: java.io.IOException:
> >> org.apache.hadoop.fs.FSError: java.io.IOException: No space left on
> device
> >> java.io.IOException: org.apache.hadoop.fs.FSError: java.io.IOException:
> No
> >> space left on device
> >> at
> >>
> >>
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> >> at
> >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> >> at
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> >> at
> >>
> >>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
> >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> at
> >>
> >>
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.writeChunk(ChecksumFileSystem.java:346)
> >> at
> >>
> >>
> org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
> >> at
> >> org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
> >> at
> org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
> >> at
> >>
> >>
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:47)
> >> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> >> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
> >> at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
> >> at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
> >> at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
> >> at java.io.BufferedWriter.close(BufferedWriter.java:248)
> >> at java.io.PrintWriter.close(PrintWriter.java:295)
> >> at
> >>
> >>
> org.apache.hadoop.mapred.JobHistory$JobInfo.logFinished(JobHistory.java:1024)
> >> at
> >>
> org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:1906)
> >> at org.apache.hadoop.mapred.JobInProgress.comp
> >>
> >>
> >>
> >> following are the disk information on dfs UI
> >> domU-12-31-39-00-0C-A1<
> >>
> http://domu-12-31-39-00-0c-a1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >0In
> >> Service413.380.8321.19391.360.2
> >> 94.672353 domU-12-31-39-00-16-F1<
> >>
> http://domu-12-31-39-00-16-f1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >1In
> >> Service413.380.4621.24391.670.11
> >> 94.752399 domU-12-31-39-00-45-71<
> >>
> http://domu-12-31-39-00-45-71.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >1In
> >> Service413.380.6421.34391.40.16
> >> 94.682303 domU-12-31-39-00-E5-D2<
> >>
> http://domu-12-31-39-00-e5-d2.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >0In
> >> Service413.380.6621.53391.180.16
> >> 94.632319 domU-12-31-39-01-64-12<
> >>
> http://domu-12-31-39-01-64-12.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >2In
> >> Service413.380.6421.24391.490.16
> >> 94.712264 domU-12-31-39-01-78-D1<
> >>
> http://domu-12-31-39-01-78-d1.compute-1.internal:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >> >0In
> >> Service413.380.4921.24391.650.12
> >> 94.741952
> >>
> >> I m using hadoop 0.19.0 and hbase 0.19.0
> >>
> >> n googling the error i came arcoss the JIRA issue
> >> http://issues.apache.org/jira/browse/HADOOP-4163
> >>
> >> which says tht its been fixed in this version. :(
> >>
> >> Has anyone else come up with this exception?
> >>
> >> how do we check the maximum capacity for usable dfs and non usable dfs.
> >> Thanks
> >> Raakhi,
> >>
> >
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
