[
https://issues.apache.org/jira/browse/SPARK-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559114#comment-14559114
]
Thomas Graves commented on SPARK-7705:
--------------------------------------
YARN will retry an application for you if it dies badly. You can't just
cleanup the staging directory on the kill or the retries will not work.
I've posted this same question on the PR but can you describe in more details
when this scenario happens and what YARN shows as the status of the
application.
> Cleanup of .sparkStaging directory fails if application is killed
> -----------------------------------------------------------------
>
> Key: SPARK-7705
> URL: https://issues.apache.org/jira/browse/SPARK-7705
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 1.3.0
> Reporter: Wilfred Spiegelenburg
> Priority: Minor
>
> When a streaming application is killed while running on YARN the
> .sparkStaging directory is not cleaned up. Setting
> spark.yarn.preserve.staging.files=false does not help and still leaves the
> files around.
> The changes in SPARK-7503 do not catch this case since there is no exception
> in the shutdown. When the application gets killed the AM gets told to
> shutdown and the shutdown hook is run but the clean up is not triggered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
