a long time ago, in Spark Summit 2013, Patrick Wendell said in his talk about performance (http://spark-summit.org/talk/wendell-understanding-the-performance-of-spark-applications/)
that, reduceByKey will be more efficient than groupByKey... he mentioned groupByKey "copies all data over network". is that still true? which one should we choice? because actually we can replace all of groupByKey with reduceByKey for example, if we want to use groupByKey on a RDD[ String, String ], to get a RDD[ String, Seq[String] ], we can also do it with reduceByKey: at first, map RDD[ String, String ] to RDD[ String, Seq[String] ] then, reduceByKey(_ ++ _) on this RDD[ String, Seq[String] ] -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/reduceByKey-to-get-all-associated-values-tp11645p11652.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
