Ryan Williams created SPARK-9197: ------------------------------------ Summary: Cached RDD partitions are lost when executors are dynamically deallocated Key: SPARK-9197 URL: https://issues.apache.org/jira/browse/SPARK-9197 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 1.4.1 Reporter: Ryan Williams
Currently, dynamic allocation cleans up executors that have not run any tasks for a certain amount of time. However, this often leads to cached RDD partitions being lost. Should dynamic allocation leave executors alone that have cached partitions? Should this be configurable? Is there any interest in code that would shuffle cached partitions around in preparation for executor-deallocation, to avoid this? Such logic could be useful in general for maintaining persisted RDDs across executor churn. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org