[ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153514#comment-14153514 ]
Matt Cheah commented on SPARK-1860: ----------------------------------- The change I am going to make is that when the cleanup task runs, it deletes an app directory inside the work directory if both the timestamp on the app directory and the timestamps on all of the app directory's files are older than the app directory retention time. If any files inside the app directory are recently modified, the app directory is not touched. Let me know if this change suffices to address this issue and I'll open a pull request. > Standalone Worker cleanup should not clean up running executors > --------------------------------------------------------------- > > Key: SPARK-1860 > URL: https://issues.apache.org/jira/browse/SPARK-1860 > Project: Spark > Issue Type: Bug > Components: Deploy > Affects Versions: 1.0.0 > Reporter: Aaron Davidson > Priority: Blocker > > The default values of the standalone worker cleanup code cleanup all > application data every 7 days. This includes jars that were added to any > executors that happen to be running for longer than 7 days, hitting streaming > jobs especially hard. > Executor's log/data folders should not be cleaned up if they're still > running. Until then, this behavior should not be enabled by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org