Hi,
I have been unsuccessfully trying to set the map output value class
different to the one reduce outputs (in 0.16.0). AFAIK the following should
do the trick:
conf.setMapOutputValueClass(FooWritable.class)
conf.setOutputValueClass(BarWritable.class)
However I kept getting exceptions saying B
>From the exception stack it appears that the map output class is correctly
set to FooWritable.class but you are trying to collect BarWritable(s) in
your map tasks.
Best,
RB
On Mon, Mar 24, 2008 at 1:22 PM, Chang Hu <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have been unsuccessfully trying to set t
Thanks Riccardo, but that's not the case. I checked and made sure it's
collecting FooWritable. In fact, from the following thread:
http://www.nabble.com/Different-output-classes-from-map-and-reducer-td15728122.html
My exception is the same as if map output value class was not set.
- Chang
Can you produce a simple, standalone example program that fails in this
way, and post it to the list? Thanks!
Doug
Code below, also attached. I put this together from the word count
example.
package edu.umd.cs.mapreduce;
import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.i
Chang Hu wrote:
Code below, also attached. I put this together from the word count
example.
The problem is with your combiner. When a combiner is specified, it
generates the final map output, since combination is a map-side
operation. Your combiner takes generated by your
mapper and gen
Thanks Doug! I am able to run the job after removing the setConbiner()
line. Does it hurt efficiency and how do I add a combiner?
- Chang
On Mon, Mar 24, 2008 at 6:26 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> Chang Hu wrote:
> > Code below, also attached. I put this together from the wor
That's simple, add a combiner that looks exactly like your reducer, but
collects IntWritable(s) as values. Btw, yours is a very good example,
because it is often forgotten that the (key,value) class types of combiners
should always match those of mappers.
RB
On Mon, Mar 24, 2008 at 3:42 PM, Chang
Good call. Thank you guys for helping me out. I'll do some experiments on
efficiency later and keep you guys updated.
- Chang
On Mon, Mar 24, 2008 at 6:51 PM, Riccardo Boscolo <[EMAIL PROTECTED]>
wrote:
> That's simple, add a combiner that looks exactly like your reducer, but
> collects IntWr
