CombinePerKey and GroupByKey

[Etienne Chauchot]

Hi all,

I'm migrating RDD pipelines to Dataset and I saw that Combine.PerKey is no more 
there in the Dataset API.  So, I
translated it to:


KeyValueGroupedDataset<K, KV<K, InputT>> groupedDataset =
    keyedDataset.groupByKey(KVHelpers.extractKey(), 
EncoderHelpers.genericEncoder());

Dataset<Tuple2<K, OutputT>> combinedDataset =
    groupedDataset.agg(
        new Aggregator<KV<K, InputT>, AccumT, OutputT>(combineFn).toColumn());

I have an interrogation regarding performance : as GroupByKey is generally less 
performant (entails shuffle and possible
OOM if a given key has a lot of data associated to it), I was wondering if the 
new spark optimizer translates such a DAG
into a combinePerKey behind the scene. In other words, is such a DAG going to 
be translated to a local (or partial I
don't know what terminology you use) combine and then a global combine to avoid 
shuffle?

Thanks

Etienne