Hi all,
I am trying to specify different key/value classes for combiner and reducer
in my task, for example, I want the
mapper to output integer==>(integer,float) pair, and then the combiner
outputs integer==>some structure. Finally
the reducer takes in integer==>some structure and output null==>integer.
However, I got the following exception:
java.io.IOException: Spill failed
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:597)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:579)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
at org.apache.hadoop.io.IntWritable.write(IntWritable.java:42)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:437)
at
edu.cmu.cs.lti.mapred.tasks.model1align.ModelOneTraining$MyMapper.map(ModelOneTraining.java:254)
at
edu.cmu.cs.lti.mapred.tasks.model1align.ModelOneTraining$MyMapper.map(ModelOneTraining.java:1)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
Caused by: java.io.IOException: wrong value class:
edu.cmu.cs.lti.mapred.io.TTableColumnWritable is not class
edu.cmu.cs.lti.mapred.io.IntFloatPairWritable
at
org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:998)
at
org.apache.hadoop.mapred.MapTask$CombineOutputCollector.collect(MapTask.java:1083)
at
edu.cmu.cs.lti.mapred.tasks.model1align.ModelOneTraining$BinCombiner.reduce(ModelOneTraining.java:433)
at
edu.cmu.cs.lti.mapred.tasks.model1align.ModelOneTraining$BinCombiner.reduce(ModelOneTraining.java:1)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:876)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:782)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1600(MapTask.java:272)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:707)
And I found nowhere to specify the combiner's input and output class. I
wonder is it possible to do so.
Best,
Qin
