[ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-30432: ---------------------------------- Target Version/s: (was: 2.4.5, 3.0.0) > reduce degree recomputation in StronglyConnectedComponents > ---------------------------------------------------------- > > Key: SPARK-30432 > URL: https://issues.apache.org/jira/browse/SPARK-30432 > Project: Spark > Issue Type: Improvement > Components: GraphX > Affects Versions: 3.0.0 > Reporter: li xiaosen > Priority: Major > > > So the computation happens every time in the do-while loop, the first time > the outer while loop executes. although just once per do-while loop after, it > seems, but It does reduce a lot of recomputation;because every time it jump > out of the do-while loop,there are no vertices have only out-degree or > in-degree,so it's no need to recompute degree to tag the vertices true. > I have done a small code proposal, because there is a problem when the pregel > executions have done, the degree no need to be recomputed. > > for example,the Email-EuAll data > set:[http://snap.stanford.edu/data/email-EuAll.html] > do-while loop execute 10 times,and the reduce logic happend 2 times;so it > would be helpful when computing StronglyConnectedComponents to reduce degree > computation. > > I created a branch in my fork: > [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] > > I hope you can consider this small code proposal. > Thank you very much, > Best regards, > xs-li -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org