[
https://issues.apache.org/jira/browse/SPARK-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangrui Meng resolved SPARK-6717.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request 11919
[https://github.com/apache/spark/pull/11919]
> Clear shuffle files after checkpointing in ALS
> ----------------------------------------------
>
> Key: SPARK-6717
> URL: https://issues.apache.org/jira/browse/SPARK-6717
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 1.4.0
> Reporter: Xiangrui Meng
> Assignee: holdenk
> Labels: als
> Fix For: 2.0.0
>
>
> In ALS iterations, we checkpoint RDDs to cut lineage and to reduce shuffle
> files. However, whether to clean shuffle files depends on the system GC,
> which may not be triggered in ALS iterations. So after checkpointing, before
> we let the RDD object go out of scope, we should clean its shuffle
> dependencies explicitly. This function could either stay inside ALS or go to
> Core.
> Without this feature, we can call System.gc() periodically to clean shuffle
> files of RDDs that went out of scope.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
