[
https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves reassigned SPARK-38910:
-------------------------------------
Assignee: angerszhu
> Clean sparkStaging dir should before unregister()
> -------------------------------------------------
>
> Key: SPARK-38910
> URL: https://issues.apache.org/jira/browse/SPARK-38910
> Project: Spark
> Issue Type: Task
> Components: YARN
> Affects Versions: 3.2.1, 3.3.0
> Reporter: angerszhu
> Assignee: angerszhu
> Priority: Minor
> Fix For: 3.4.0
>
>
> {code:java}
> ShutdownHookManager.addShutdownHook(priority) { () =>
> try {
> val maxAppAttempts = client.getMaxRegAttempts(sparkConf, yarnConf)
> val isLastAttempt = appAttemptId.getAttemptId() >= maxAppAttempts
> if (!finished) {
> // The default state of ApplicationMaster is failed if it is
> invoked by shut down hook.
> // This behavior is different compared to 1.x version.
> // If user application is exited ahead of time by calling
> System.exit(N), here mark
> // this application as failed with EXIT_EARLY. For a good
> shutdown, user shouldn't call
> // System.exit(0) to terminate the application.
> finish(finalStatus,
> ApplicationMaster.EXIT_EARLY,
> "Shutdown hook called before final status was reported.")
> }
> if (!unregistered) {
> // we only want to unregister if we don't want the RM to retry
> if (finalStatus == FinalApplicationStatus.SUCCEEDED ||
> isLastAttempt) {
> unregister(finalStatus, finalMsg)
> cleanupStagingDir(new
> Path(System.getenv("SPARK_YARN_STAGING_DIR")))
> }
> }
> } catch {
> case e: Throwable =>
> logWarning("Ignoring Exception while stopping ApplicationMaster
> from shutdown hook", e)
> }
> }{code}
> unregister may throw exception, clean staging dir should before unregister.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
