What happens if I have a function like a PairFunction but which might return multiple values

2014-08-28 Thread Steve Lewis
In many cases when I work with Map Reduce my mapper or my reducer might take a single value and map it to multiple keys - The reducer might also take a single key and emit multiple values I don't think that functions like flatMap and reduceByKey will work or are there tricks I am not aware of

Re: What happens if I have a function like a PairFunction but which might return multiple values

2014-08-28 Thread Sean Owen
To emulate a Mapper, flatMap() is exactly what you want. Since it flattens, it means you return an Iterable of values instead of 1 value. That can be a Collection containing many values, or 1, or 0. For a reducer, to really reproduce what a Reducer does in Java, I think you will need groupByKey()