[
https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
li xiaosen updated SPARK-30432:
-------------------------------
Fix Version/s: (was: 2.4.4)
Target Version/s: 2.4.5, 3.0.0 (was: 2.4.4, 2.4.5)
Affects Version/s: (was: 2.4.4)
Description:
So the computation happens every time in the do-while loop, the first time the
outer while loop executes. although just once per do-while loop after, it
seems, but It does reduce a lot of recomputation；because every time it jump out
of the do-while loop，there are no vertices have only out-degree or in-degree，so
it's no need to recompute degree to tag the vertices true.
I have done a small code proposal, because there is a problem when the pregel
executions have done, the degree no need to be recomputed.
for example，the Email-EuAll data
set：[http://snap.stanford.edu/data/email-EuAll.html]
do-while loop execute 10 times，and the reduce logic happend 8 times；so it would
be helpful when computing StronglyConnectedComponents to reduce degree
computation.
I created a branch in my fork:
[https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala]
I hope you can consider this small code proposal.
Thank you very much,
Best regards,
xs-li
was:
It would be helpful when computing StronglyConnectedComponents to reduce degree
computation.
I have done a small code proposal, because there is a problem when the pregel
executions have done, the degree no need to be recomputed.
I created a branch in my fork:
[https://github.com/xs-li/spark/blob/branch-2.4/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala]
I hope you can consider this small code proposal.
Thank you very much,
Best regards,
xs-li
Priority: Major (was: Minor)
> reduce degree recomputation in StronglyConnectedComponents
> ----------------------------------------------------------
>
> Key: SPARK-30432
> URL: https://issues.apache.org/jira/browse/SPARK-30432
> Project: Spark
> Issue Type: Improvement
> Components: GraphX
> Affects Versions: 2.4.5, 3.0.0
> Reporter: li xiaosen
> Priority: Major
>
>
> So the computation happens every time in the do-while loop, the first time
> the outer while loop executes. although just once per do-while loop after, it
> seems, but It does reduce a lot of recomputation；because every time it jump
> out of the do-while loop，there are no vertices have only out-degree or
> in-degree，so it's no need to recompute degree to tag the vertices true.
> I have done a small code proposal, because there is a problem when the pregel
> executions have done, the degree no need to be recomputed.
>
> for example，the Email-EuAll data
> set：[http://snap.stanford.edu/data/email-EuAll.html]
> do-while loop execute 10 times，and the reduce logic happend 8 times；so it
> would be helpful when computing StronglyConnectedComponents to reduce degree
> computation.
>
> I created a branch in my fork:
> [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala]
>
> I hope you can consider this small code proposal.
> Thank you very much,
> Best regards,
> xs-li
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
