Hi there, A while ago running GraphX jobs I've discovered that PeriodicRDDCheckpointer fails with FileNotFoundException's in case of insufficient memory resources.
I believe that any iterative job which uses PeriodicRDDCheckpointer (like ML) suffers from the same issue (but it's not visible enough because of RAM size of modern servers). So, I've raised the JIRA issues with the corresponding pull requests to fix them - https://issues.apache.org/jira/browse/SPARK-22150 - https://issues.apache.org/jira/browse/SPARK-22184 - https://github.com/apache/spark/pull/19373 - https://github.com/apache/spark/pull/19410 Could anyone please look through these PRs? --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
