Repository: spark Updated Branches: refs/heads/branch-2.4 b632e775c -> 085f731ad
[SPARK-25268][GRAPHX] run Parallel Personalized PageRank throws serialization Exception ## What changes were proposed in this pull request? mapValues in scala is currently not serializable. To avoid the serialization issue while running pageRank, we need to use map instead of mapValues. Please review http://spark.apache.org/contributing.html before opening a pull request. Closes #22271 from shahidki31/master_latest. Authored-by: Shahid <shahidk...@gmail.com> Signed-off-by: Joseph K. Bradley <jos...@databricks.com> (cherry picked from commit 3b6591b0b064b13a411e5b8f8ee4883a69c39e2d) Signed-off-by: Joseph K. Bradley <jos...@databricks.com> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/085f731a Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/085f731a Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/085f731a Branch: refs/heads/branch-2.4 Commit: 085f731adb9b8c82a2bf4bbcae6d889a967fbd53 Parents: b632e77 Author: Shahid <shahidk...@gmail.com> Authored: Thu Sep 6 09:52:58 2018 -0700 Committer: Joseph K. Bradley <jos...@databricks.com> Committed: Thu Sep 6 09:53:07 2018 -0700 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/graphx/lib/PageRank.scala | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/085f731a/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala ---------------------------------------------------------------------- diff --git a/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala b/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala index 96b635f..1305c05 100644 --- a/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala +++ b/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala @@ -198,9 +198,11 @@ object PageRank extends Logging { val zero = Vectors.sparse(sources.size, List()).asBreeze // map of vid -> vector where for each vid, the _position of vid in source_ is set to 1.0 - val sourcesInitMap = sources.zipWithIndex.toMap.mapValues { i => - Vectors.sparse(sources.size, Array(i), Array(1.0)).asBreeze - } + val sourcesInitMap = sources.zipWithIndex.map { case (vid, i) => + val v = Vectors.sparse(sources.size, Array(i), Array(1.0)).asBreeze + (vid, v) + }.toMap + val sc = graph.vertices.sparkContext val sourcesInitMapBC = sc.broadcast(sourcesInitMap) // Initialize the PageRank graph with each edge attribute having --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org