[jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents
[ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30432: -- Affects Version/s: (was: 2.4.5) (was: 3.0.0) 3.1.0 > reduce degree recomputation in StronglyConnectedComponents > -- > > Key: SPARK-30432 > URL: https://issues.apache.org/jira/browse/SPARK-30432 > Project: Spark > Issue Type: Improvement > Components: GraphX >Affects Versions: 3.1.0 >Reporter: xiaosen >Priority: Major > > > So the computation happens every time in the do-while loop, the first time > the outer while loop executes. although just once per do-while loop after, it > seems, but It does reduce a lot of recomputation；because every time it jump > out of the do-while loop，there are no vertices have only out-degree or > in-degree，so it's no need to recompute degree to tag the vertices true. > I have done a small code proposal, because there is a problem when the pregel > executions have done, the degree no need to be recomputed. > > for example，the Email-EuAll data > set：[http://snap.stanford.edu/data/email-EuAll.html] > do-while loop execute 10 times，and the reduce logic happend 2 times；so it > would be helpful when computing StronglyConnectedComponents to reduce degree > computation. > > I created a branch in my fork: > [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] > > I hope you can consider this small code proposal. > Thank you very much, > Best regards, > xs-li -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents
[ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] li xiaosen updated SPARK-30432: --- Affects Version/s: 2.4.5 > reduce degree recomputation in StronglyConnectedComponents > -- > > Key: SPARK-30432 > URL: https://issues.apache.org/jira/browse/SPARK-30432 > Project: Spark > Issue Type: Improvement > Components: GraphX >Affects Versions: 2.4.5, 3.0.0 >Reporter: li xiaosen >Priority: Major > > > So the computation happens every time in the do-while loop, the first time > the outer while loop executes. although just once per do-while loop after, it > seems, but It does reduce a lot of recomputation；because every time it jump > out of the do-while loop，there are no vertices have only out-degree or > in-degree，so it's no need to recompute degree to tag the vertices true. > I have done a small code proposal, because there is a problem when the pregel > executions have done, the degree no need to be recomputed. > > for example，the Email-EuAll data > set：[http://snap.stanford.edu/data/email-EuAll.html] > do-while loop execute 10 times，and the reduce logic happend 2 times；so it > would be helpful when computing StronglyConnectedComponents to reduce degree > computation. > > I created a branch in my fork: > [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] > > I hope you can consider this small code proposal. > Thank you very much, > Best regards, > xs-li -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents
[ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30432: -- Target Version/s: (was: 2.4.5, 3.0.0) > reduce degree recomputation in StronglyConnectedComponents > -- > > Key: SPARK-30432 > URL: https://issues.apache.org/jira/browse/SPARK-30432 > Project: Spark > Issue Type: Improvement > Components: GraphX >Affects Versions: 3.0.0 >Reporter: li xiaosen >Priority: Major > > > So the computation happens every time in the do-while loop, the first time > the outer while loop executes. although just once per do-while loop after, it > seems, but It does reduce a lot of recomputation；because every time it jump > out of the do-while loop，there are no vertices have only out-degree or > in-degree，so it's no need to recompute degree to tag the vertices true. > I have done a small code proposal, because there is a problem when the pregel > executions have done, the degree no need to be recomputed. > > for example，the Email-EuAll data > set：[http://snap.stanford.edu/data/email-EuAll.html] > do-while loop execute 10 times，and the reduce logic happend 2 times；so it > would be helpful when computing StronglyConnectedComponents to reduce degree > computation. > > I created a branch in my fork: > [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] > > I hope you can consider this small code proposal. > Thank you very much, > Best regards, > xs-li -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents
[ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30432: -- Affects Version/s: (was: 2.4.5) > reduce degree recomputation in StronglyConnectedComponents > -- > > Key: SPARK-30432 > URL: https://issues.apache.org/jira/browse/SPARK-30432 > Project: Spark > Issue Type: Improvement > Components: GraphX >Affects Versions: 3.0.0 >Reporter: li xiaosen >Priority: Major > > > So the computation happens every time in the do-while loop, the first time > the outer while loop executes. although just once per do-while loop after, it > seems, but It does reduce a lot of recomputation；because every time it jump > out of the do-while loop，there are no vertices have only out-degree or > in-degree，so it's no need to recompute degree to tag the vertices true. > I have done a small code proposal, because there is a problem when the pregel > executions have done, the degree no need to be recomputed. > > for example，the Email-EuAll data > set：[http://snap.stanford.edu/data/email-EuAll.html] > do-while loop execute 10 times，and the reduce logic happend 2 times；so it > would be helpful when computing StronglyConnectedComponents to reduce degree > computation. > > I created a branch in my fork: > [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] > > I hope you can consider this small code proposal. > Thank you very much, > Best regards, > xs-li -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents
[ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] li xiaosen updated SPARK-30432: --- Shepherd: (was: Saisai Shao) > reduce degree recomputation in StronglyConnectedComponents > -- > > Key: SPARK-30432 > URL: https://issues.apache.org/jira/browse/SPARK-30432 > Project: Spark > Issue Type: Improvement > Components: GraphX >Affects Versions: 2.4.5, 3.0.0 >Reporter: li xiaosen >Priority: Major > > > So the computation happens every time in the do-while loop, the first time > the outer while loop executes. although just once per do-while loop after, it > seems, but It does reduce a lot of recomputation；because every time it jump > out of the do-while loop，there are no vertices have only out-degree or > in-degree，so it's no need to recompute degree to tag the vertices true. > I have done a small code proposal, because there is a problem when the pregel > executions have done, the degree no need to be recomputed. > > for example，the Email-EuAll data > set：[http://snap.stanford.edu/data/email-EuAll.html] > do-while loop execute 10 times，and the reduce logic happend 2 times；so it > would be helpful when computing StronglyConnectedComponents to reduce degree > computation. > > I created a branch in my fork: > [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] > > I hope you can consider this small code proposal. > Thank you very much, > Best regards, > xs-li -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents
[ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] li xiaosen updated SPARK-30432: --- Description: So the computation happens every time in the do-while loop, the first time the outer while loop executes. although just once per do-while loop after, it seems, but It does reduce a lot of recomputation；because every time it jump out of the do-while loop，there are no vertices have only out-degree or in-degree，so it's no need to recompute degree to tag the vertices true. I have done a small code proposal, because there is a problem when the pregel executions have done, the degree no need to be recomputed. for example，the Email-EuAll data set：[http://snap.stanford.edu/data/email-EuAll.html] do-while loop execute 10 times，and the reduce logic happend 2 times；so it would be helpful when computing StronglyConnectedComponents to reduce degree computation. I created a branch in my fork: [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] I hope you can consider this small code proposal. Thank you very much, Best regards, xs-li was: So the computation happens every time in the do-while loop, the first time the outer while loop executes. although just once per do-while loop after, it seems, but It does reduce a lot of recomputation；because every time it jump out of the do-while loop，there are no vertices have only out-degree or in-degree，so it's no need to recompute degree to tag the vertices true. I have done a small code proposal, because there is a problem when the pregel executions have done, the degree no need to be recomputed. for example，the Email-EuAll data set：[http://snap.stanford.edu/data/email-EuAll.html] do-while loop execute 10 times，and the reduce logic happend 8 times；so it would be helpful when computing StronglyConnectedComponents to reduce degree computation. I created a branch in my fork: [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] I hope you can consider this small code proposal. Thank you very much, Best regards, xs-li > reduce degree recomputation in StronglyConnectedComponents > -- > > Key: SPARK-30432 > URL: https://issues.apache.org/jira/browse/SPARK-30432 > Project: Spark > Issue Type: Improvement > Components: GraphX >Affects Versions: 2.4.5, 3.0.0 >Reporter: li xiaosen >Priority: Major > > > So the computation happens every time in the do-while loop, the first time > the outer while loop executes. although just once per do-while loop after, it > seems, but It does reduce a lot of recomputation；because every time it jump > out of the do-while loop，there are no vertices have only out-degree or > in-degree，so it's no need to recompute degree to tag the vertices true. > I have done a small code proposal, because there is a problem when the pregel > executions have done, the degree no need to be recomputed. > > for example，the Email-EuAll data > set：[http://snap.stanford.edu/data/email-EuAll.html] > do-while loop execute 10 times，and the reduce logic happend 2 times；so it > would be helpful when computing StronglyConnectedComponents to reduce degree > computation. > > I created a branch in my fork: > [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] > > I hope you can consider this small code proposal. > Thank you very much, > Best regards, > xs-li -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-30432) reduce degree recomputation in StronglyConnectedComponents
[ https://issues.apache.org/jira/browse/SPARK-30432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] li xiaosen updated SPARK-30432: --- Fix Version/s: (was: 2.4.4) Target Version/s: 2.4.5, 3.0.0 (was: 2.4.4, 2.4.5) Affects Version/s: (was: 2.4.4) Description: So the computation happens every time in the do-while loop, the first time the outer while loop executes. although just once per do-while loop after, it seems, but It does reduce a lot of recomputation；because every time it jump out of the do-while loop，there are no vertices have only out-degree or in-degree，so it's no need to recompute degree to tag the vertices true. I have done a small code proposal, because there is a problem when the pregel executions have done, the degree no need to be recomputed. for example，the Email-EuAll data set：[http://snap.stanford.edu/data/email-EuAll.html] do-while loop execute 10 times，and the reduce logic happend 8 times；so it would be helpful when computing StronglyConnectedComponents to reduce degree computation. I created a branch in my fork: [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] I hope you can consider this small code proposal. Thank you very much, Best regards, xs-li was: It would be helpful when computing StronglyConnectedComponents to reduce degree computation. I have done a small code proposal, because there is a problem when the pregel executions have done, the degree no need to be recomputed. I created a branch in my fork: [https://github.com/xs-li/spark/blob/branch-2.4/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] I hope you can consider this small code proposal. Thank you very much, Best regards, xs-li Priority: Major (was: Minor) > reduce degree recomputation in StronglyConnectedComponents > -- > > Key: SPARK-30432 > URL: https://issues.apache.org/jira/browse/SPARK-30432 > Project: Spark > Issue Type: Improvement > Components: GraphX >Affects Versions: 2.4.5, 3.0.0 >Reporter: li xiaosen >Priority: Major > > > So the computation happens every time in the do-while loop, the first time > the outer while loop executes. although just once per do-while loop after, it > seems, but It does reduce a lot of recomputation；because every time it jump > out of the do-while loop，there are no vertices have only out-degree or > in-degree，so it's no need to recompute degree to tag the vertices true. > I have done a small code proposal, because there is a problem when the pregel > executions have done, the degree no need to be recomputed. > > for example，the Email-EuAll data > set：[http://snap.stanford.edu/data/email-EuAll.html] > do-while loop execute 10 times，and the reduce logic happend 8 times；so it > would be helpful when computing StronglyConnectedComponents to reduce degree > computation. > > I created a branch in my fork: > [https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala] > > I hope you can consider this small code proposal. > Thank you very much, > Best regards, > xs-li -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
