[ https://issues.apache.org/jira/browse/SPARK-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-9197. ------------------------------ Resolution: Duplicate > Cached RDD partitions are lost when executors are dynamically deallocated > ------------------------------------------------------------------------- > > Key: SPARK-9197 > URL: https://issues.apache.org/jira/browse/SPARK-9197 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 1.4.1 > Reporter: Ryan Williams > Priority: Minor > > Currently, dynamic allocation cleans up executors that have not run any tasks > for a certain amount of time. > However, this often leads to cached RDD partitions being lost. > Should dynamic allocation leave executors alone that have cached partitions? > Should this be configurable? > Is there any interest in code that would shuffle cached partitions around in > preparation for executor-deallocation, to avoid this? Such logic could be > useful in general for maintaining persisted RDDs across executor churn. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org