[
https://issues.apache.org/jira/browse/SPARK-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Shar updated SPARK-18027:
-------------------------------
Summary: .sparkStaging not clean on RM ApplicationNotFoundException (was:
.sparkStaging not clean on error)
> .sparkStaging not clean on RM ApplicationNotFoundException
> ----------------------------------------------------------
>
> Key: SPARK-18027
> URL: https://issues.apache.org/jira/browse/SPARK-18027
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.6.0
> Reporter: David Shar
>
> Hi,
> It seems that SPARK-7705 didn't fix all issues with .sparkStaging folder
> cleanup.
> in Client.scala:monitorApplication
> {code}
> val report: ApplicationReport =
> try {
> getApplicationReport(appId)
> } catch {
> case e: ApplicationNotFoundException =>
> logError(s"Application $appId not found.")
> return (YarnApplicationState.KILLED,
> FinalApplicationStatus.KILLED)
> case NonFatal(e) =>
> logError(s"Failed to contact YARN for application $appId.", e)
> return (YarnApplicationState.FAILED,
> FinalApplicationStatus.FAILED)
> }
> ....
> if (state == YarnApplicationState.FINISHED ||
> state == YarnApplicationState.FAILED ||
> state == YarnApplicationState.KILLED) {
> cleanupStagingDir(appId)
> return (state, report.getFinalApplicationStatus)
> }
> {code}
> In case of ApplicationNotFoundException, we don't cleanup the sparkStaging
> folder.
> I believe we should call cleanupStagingDir(appId) on the catch clause above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
