Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/19317#discussion_r144760426 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -180,6 +180,56 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)]) * as in scala.TraversableOnce. The former operation is used for merging values within a * partition, and the latter is used for merging values between partitions. To avoid memory * allocation, both of these functions are allowed to modify and return their first argument + * instead of creating a new U. This method is different from the ordinary "aggregateByKey" + * method, it directly returns a map to the driver, rather than a rdd. This will also perform + * the merging locally on each mapper before sending results to a reducer, similarly to a --- End diff -- `aggregateByKey(...).toLocalIterator` need a shuffle for `aggregateByKey` and then collect the `RDD` to driver as a iterator. But `aggregateByKeyLocally` seems like the `aggregateByKey`, while there isn't a shuffle. It calculates the combines in each task and then collect all the `map` direcly to driver and do the finally combines on driver.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org