Thanks Runping. It seems the bug is still open. However, in my case, there were more than plenty of disk space available.
On Jan 16, 2008 2:44 AM, Runping Qi <[EMAIL PROTECTED]> wrote: > > I encountered a similar case. > Here is the Jira: https://issues.apache.org/jira/browse/HADOOP-2164 > > Runping > > > > > -----Original Message----- > > From: Vadim Zaliva [mailto:[EMAIL PROTECTED] > > Sent: Tuesday, January 15, 2008 9:59 PM > > To: hadoop-user@lucene.apache.org > > Subject: Re: unable to figure out this exception from reduce task > > > > On Jan 15, 2008, at 21:53, Jim the Standing Bear wrote: > > > > I was asking lot of questions today, so I am glad to contribute at > > least one answer. I have this problem when there was type mismatch > > for key or values. You need to set up right type at your JobConf like > > this: > > > > conf.setOutputKeyClass(Text.class); > > conf.setOutputValueClass(LongWritable.class); > > > > (using appropriate types our mapper produce) > > > > Vadim > > > > > I am using hadoop 0.15.1 to index some catalog that has a tree-like > > > structure, where the leaf nodes are data files. My main task is a > > > loop that performs a breadth-first walkthrough that parses out URLs > to > > > catalogs and datafiles at the next level, which is done in a mapper. > > > To determine when the loop should terminate, I use a reduce task > that > > > counts the number of new catalogs found, and stops the loop when the > > > count is 0. > > > > > > But while I was running the jobs, I kept getting this exception > > > (pasted below from the logs). I didn't quite understand what it was > > > trying to say. But in my code, I never used LongWritable. Only > Text > > > for output key and output values, and KeyValueTextInputFormat for > > > input. > > > > > > What's weirder is that this exception occurs at different places > from > > > job to job. Sometimes it may be thrown at the 2nd iteration of my > > > loop, while other times, it may be the 3rd, the 4th etc. Can > someone > > > explain to me what and why this is? Also, what would be the best > way > > > to test/debug a hadoop job?? Thanks. > > > > > > > > > 2008-01-16 00:37:19,941 INFO org.apache.hadoop.mapred.ReduceTask: > > > task_200801160024_0011_r_000000_1 Copying > > > task_200801160024_0011_m_000000_0 output from ginkgo.mycluster.org > > > 2008-01-16 00:37:19,953 INFO org.apache.hadoop.mapred.ReduceTask: > > > task_200801160024_0011_r_000000_1 done copying > > > task_200801160024_0011_m_000000_0 output from ginkgo.mycluster.org > > > 2008-01-16 00:37:19,955 INFO org.apache.hadoop.mapred.ReduceTask: > > > task_200801160024_0011_r_000000_1 Copying of all map outputs > complete. > > > Initiating the last merge on the remaining files in > > > ramfs://mapoutput26453615 > > > 2008-01-16 00:37:20,088 WARN org.apache.hadoop.mapred.ReduceTask: > > > task_200801160024_0011_r_000000_1 Final merge of the inmemory files > > > threw an exception: java.io.IOException: java.io.IOException: wrong > > > key class: class org.apache.hadoop.io.LongWritable is not class > > > org.apache.hadoop.io.Text > > > at org.apache.hadoop.io.SequenceFile$Sorter > > > $SegmentDescriptor.nextRawKey(SequenceFile.java:2874) > > > at org.apache.hadoop.io.SequenceFile$Sorter > > > $MergeQueue.merge(SequenceFile.java:2683) > > > at > org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java: > > > 2437) > > > at org.apache.hadoop.mapred.ReduceTask > > > $ReduceCopier.fetchOutputs(ReduceTask.java:1153) > > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:252) > > > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java: > > > 1760) > > > > > > at org.apache.hadoop.mapred.ReduceTask > > > $ReduceCopier.fetchOutputs(ReduceTask.java:1161) > > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:252) > > > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java: > > > 1760) > > > > > > 2008-01-16 00:37:20,090 WARN org.apache.hadoop.mapred.TaskTracker: > > > Error running child > > > java.io.IOException: task_200801160024_0011_r_000000_1The reduce > > > copier failed > > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:253) > > > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java: > > > 1760) > > > > > > > > > > > > -- > > > -------------------------------------- > > > Standing Bear Has Spoken > > > -------------------------------------- > > -- -------------------------------------- Standing Bear Has Spoken --------------------------------------
