If the map-side-combine is not that necessary, given the fact that it cannot
reduce the size of data for shuffling much (do need to serialized the key for
each value), but can reduce the number of key-value pairs, and potential reduce
the number of operations later (repartition and groupby).
On Tue, Jul 14, 2015 at 7:11 PM, Matt Cheah mch...@palantir.com wrote:
Hi everyone,
I was examining the Pyspark implementation of groupByKey in rdd.py. I would
like to submit a patch improving Scala RDD’s groupByKey that has a similar
robustness against large groups, as Pyspark’s implementation has logic to
spill part of a single group to disk along the way.
Its implementation appears to do the following:
Combine and group-by-key per partition locally, potentially spilling
individual groups to disk
Shuffle the data explicitly using partitionBy
After the shuffle, do another local groupByKey to get the final result,
again potentially spilling individual groups to disk
My question is: what does the explicit map-side-combine step (#1)
specifically benefit here? I was under the impression that map-side-combine
for groupByKey was not optimal and is turned off in the Scala implementation
– Scala PairRDDFunctions.groupByKey calls to combineByKey with
map-side-combine set to false. Is it something specific to how Pyspark can
potentially spill the individual groups to disk?
Thanks,
-Matt Cheah
P.S. Relevant Links:
https://issues.apache.org/jira/browse/SPARK-3074
https://github.com/apache/spark/pull/1977
-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
