check that you are not getting disk full errors Miles
2008/9/29 Elia Mazzawi <[EMAIL PROTECTED]>: > in more detail, my program is happily chugging along until the reducer fails > with that exception, then it looks like it retries and fails by itself. > the same hadoop program works fine on a subset of the data. > I'm rerunning on all the subsets to see if there is anything in the data > that is causing this, > > but can someone explain what this error means, > I'm runnig hadoop 17.0 maybe its time to update > > > 08/09/27 05:59:01 INFO mapred.JobClient: map 96% reduce 31% > 08/09/27 06:02:17 INFO mapred.JobClient: map 96% reduce 32% > 08/09/27 06:24:46 INFO mapred.JobClient: map 97% reduce 32% > 08/09/27 06:49:38 INFO mapred.JobClient: map 98% reduce 32% > 08/09/27 07:14:12 INFO mapred.JobClient: map 99% reduce 32% > 08/09/27 07:17:09 INFO mapred.JobClient: map 99% reduce 33% > 08/09/27 07:37:50 INFO mapred.JobClient: map 100% reduce 33% > 08/09/27 07:56:11 INFO mapred.JobClient: map 100% reduce 0% > 08/09/27 07:56:11 INFO mapred.JobClient: Task Id : > task_200809041356_0042_r_000000_2, Status : FAILED > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any > valid local directory for task_200809041356_0042_r_000000_2/intermediate.9 > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) > at > org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2851) > at > org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2586) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:352) > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124) > > 08/09/27 07:57:10 INFO mapred.JobClient: map 100% reduce 1% > 08/09/27 07:57:55 INFO mapred.JobClient: map 100% reduce 2% > 08/09/27 07:58:46 INFO mapred.JobClient: map 100% reduce 3% > 08/09/27 07:59:36 INFO mapred.JobClient: map 100% reduce 4% > 08/09/27 08:00:26 INFO mapred.JobClient: map 100% reduce 5% > 08/09/27 08:01:16 INFO mapred.JobClient: map 100% reduce 6% > 08/09/27 08:02:06 INFO mapred.JobClient: map 100% reduce 7% > 08/09/27 08:02:55 INFO mapred.JobClient: map 100% reduce 8% > 08/09/27 08:03:45 INFO mapred.JobClient: map 100% reduce 9% > 08/09/27 08:04:36 INFO mapred.JobClient: map 100% reduce 10% > 08/09/27 08:05:26 INFO mapred.JobClient: map 100% reduce 11% > 08/09/27 08:06:18 INFO mapred.JobClient: map 100% reduce 12% > 08/09/27 08:07:09 INFO mapred.JobClient: map 100% reduce 13% > 08/09/27 08:08:00 INFO mapred.JobClient: map 100% reduce 14% > 08/09/27 08:08:50 INFO mapred.JobClient: map 100% reduce 15% > 08/09/27 08:09:45 INFO mapred.JobClient: map 100% reduce 16% > 08/09/27 08:10:31 INFO mapred.JobClient: map 100% reduce 17% > 08/09/27 08:11:26 INFO mapred.JobClient: map 100% reduce 18% > 08/09/27 08:12:16 INFO mapred.JobClient: map 100% reduce 19% > 08/09/27 08:13:08 INFO mapred.JobClient: map 100% reduce 20% > 08/09/27 08:14:02 INFO mapred.JobClient: map 100% reduce 21% > 08/09/27 08:14:50 INFO mapred.JobClient: map 100% reduce 22% > 08/09/27 08:15:41 INFO mapred.JobClient: map 100% reduce 23% > 08/09/27 08:16:36 INFO mapred.JobClient: map 100% reduce 24% > 08/09/27 08:17:26 INFO mapred.JobClient: map 100% reduce 25% > 08/09/27 08:18:14 INFO mapred.JobClient: map 100% reduce 26% > 08/09/27 08:19:02 INFO mapred.JobClient: map 100% reduce 27% > 08/09/27 08:19:55 INFO mapred.JobClient: map 100% reduce 28% > 08/09/27 08:21:12 INFO mapred.JobClient: map 100% reduce 71% > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062) > at org.myorg.binAnalysis.main(binAnalysis.java:99) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at org.apache.hadoop.util.RunJar.main(RunJar.java:155) > at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220) > > You have new mail in /var/spool/mail/root > > > Elia Mazzawi wrote: >> >> what does this exception mean? >> >> 08/09/27 07:56:11 INFO mapred.JobClient: Task Id : >> task_200809041356_0042_r_000000_2, Status : FAILED >> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any >> valid local directory for task_200809041356_0042_r_000000_2/intermediate.9 >> at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313) >> at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) >> at >> org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2851) >> at >> org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2586) >> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:352) >> at >> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124) >> > > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
