[
https://issues.apache.org/jira/browse/HADOOP-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893523#action_12893523
]
Meng Mao commented on HADOOP-6092:
----------------------------------
one more comment --
we have a nightly process composed of Java jobs, that runs over roughly the
same data set size each time. One of the big upfront filtering jobs seems to
currently the most at-risk for triggering a No space left error.
As noted above, we have about 13TB free. As this number shrinks past 10TB, the
incidence of No space left errors rises. This is only empirical, but we try to
keep more than 10TB free just to avoid it.
Without any knowledge of what's going on internally, it seems that there's
certainly correlation between HDFS space left and likelihood of a job on
roughly the same input size failing with No space left. What's puzzling us is
why it's when there's still as much as 10TB free, for a job that's on the order
of 100GB of input.
> No space left on device
> -----------------------
>
> Key: HADOOP-6092
> URL: https://issues.apache.org/jira/browse/HADOOP-6092
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 0.19.0
> Environment: ubuntu0.8.4
> Reporter: mawanqiang
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException:
> No space left on device
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:339)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:825)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
> at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
> at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
> Caused by: java.io.IOException: No space left on device
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
