Hello there,
I got the following errors relative to disk problem. I checked the
slave node which runs the task, only half of the disk space is used. I
don't understand why that happens. The application I run dumps most of
the intermediate keys to one particular reducer so that particular
reducer has to take a large amount of data. I intentionally do that
for key partition investigation. Any suggestion would be appreciated!
I tried to adjust several parameters
mapred.job.reduce.input.buffer.percent=0.9
io.sort.spill.percent=0.9
io.sort.mb=1024
io.sort.factor=80
FSError: java.io.IOException: No space left on device
java.io.IOException: Task: attempt_201202172329_0022_r_000000_1 - The
reduce copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
Could not find any valid local directory for
taskTracker/jobcache/job_201202172329_0022/attempt_201202172329_0022_r_000000_1/output/map_10.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputFile.java:160)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2537)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2501)
java.io.IOException: Task: attempt_201202172329_0038_r_000000_0 - The
reduce copier failed
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException:
Could not find any valid local directory for
taskTracker/jobcache/job_201202172329_0038/attempt_201202172329_0038_r_000000_0/output/map_1.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputFile.java:160)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2537)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2501)
--
Allen
