GitHub user szhem opened a pull request: https://github.com/apache/spark/pull/19410
[SPARK-22184][CORE][GRAPHX] GraphX fails in case of insufficient memory and checkpoints enabled ## What changes were proposed in this pull request? Fix for [SPARK-22184](https://issues.apache.org/jira/browse/SPARK-22184) JIRA issue (and also includes the related #19373). In case of GraphX jobs, when checkpoints are enabled, GraphX can fail with `FileNotFoundException`. The failure can happen during Pregel iterations or when Pregel completes only in cases of insufficient memory when checkpointed RDDs are evicted from memory and have to be read from disk (but already removed from there). This PR proposes to preserve all the checkpoints the last one (checkpoint) of `messages` and `graph` depends on during the iterations, and all the checkpoints of `messages` and `graph` the resulting `graph` depends at the end of Pregel iterations. ## How was this patch tested? Unit tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/szhem/spark SPARK-22184-graphx-early-checkpoints-removal Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19410.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19410 ---- commit f2386b61a47abf19b8ca6cea7e0e5c7da9baf7d6 Author: Sergey Zhemzhitsky <szhemzhit...@gmail.com> Date: 2017-09-27T21:33:18Z [SPARK-22150][CORE] preventing too early removal of checkpoints in case of dependant RDDs commit aa2bedae74999694b0a9992986e85d3f9feab5f6 Author: Sergey Zhemzhitsky <szhemzhit...@gmail.com> Date: 2017-10-02T13:10:48Z [SPARK-22150][CORE] checking whether two checkpoints have the same checkpointed RDD as their parent to prevent early removal commit 6406aea3bc87c1f3a9460bbc2ae1af67d7c0c294 Author: Sergey Zhemzhitsky <szhemzhit...@gmail.com> Date: 2017-10-02T13:22:19Z [SPARK-22150][CORE] respecting scala style settings commit 4a55cda79e61e7eec67ae9545beb0c38eca7b11b Author: Sergey Zhemzhitsky <szhemzhit...@gmail.com> Date: 2017-10-02T14:43:27Z [SPARK-22184][CORE][GRAPHX] retain all the checkpoints the last one depends on ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org