GitHub user wesolowskim opened a pull request:
https://github.com/apache/spark/pull/14137
SPARK-16478 graphX (added graph caching in strongly connected components)
## What changes were proposed in this pull request?
I added caching in every iteration for sccGraph that is returned in
strongly connected components. Without this cache strongly connected components
returned graph that needed to be computed from scratch when some intermediary
caches didn't existed anymore.
## How was this patch tested?
I tested it by running code similar to the one [on
databrics](https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/4889410027417133/3634650767364730/3117184429335832/latest.html).
Basically I generated large graph and computed strongly connected components
with changed code, than simply run count on vertices and edges. Count after
this update takes few seconds instead 20 minutes.
# statement
contribution is my original work and I license the work to the project
under the project's open source license.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wesolowskim/spark SPARK-16478
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14137.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14137
----
commit f9e28830755535926f7792375cedadb0518ecb31
Author: MichaÅ WesoÅowski <michal.wesolow...@bzwbk.pl>
Date: 2016-07-11T11:20:26Z
corrected SPARK-16478 (caching in strongly connected components)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
