Combiners in Spark

2015-03-02 Thread Guillermo Ortiz
Which is the equivalent function to Combiners of MapReduce in Spark? I guess that it's combineByKey, but is combineByKey executed locally? I understand than functions as reduceByKey or foldByKey aren't executed locally. Reading the documentation looks like combineByKey is equivalent to

Re: Combiners in Spark

2015-03-02 Thread Sean Owen
I think the simplest answer is that it's not really a separate concept from the 'reduce' function, because Spark's API is a sort of simpler, purer form of FP. It is just the same function that can be applied at many points in an aggregation -- both map side (a la Combiners in MapReduce) or reduce