This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 87c744b6050 [SPARK-38910][YARN] Clean spark staging before `unregister`
87c744b6050 is described below
commit 87c744b60507f82e1722f1488f1741cb2bb8e8e5
Author: Angerszhuuuu <angers....@gmail.com>
AuthorDate: Mon Apr 18 11:18:48 2022 -0500
[SPARK-38910][YARN] Clean spark staging before `unregister`
### What changes were proposed in this pull request?
`ApplicationMaster`'s shutdown
```
ShutdownHookManager.addShutdownHook(priority) { () =>
try {
val maxAppAttempts = client.getMaxRegAttempts(sparkConf, yarnConf)
val isLastAttempt = appAttemptId.getAttemptId() >= maxAppAttempts
if (!finished) {
// The default state of ApplicationMaster is failed if it is
invoked by shut down hook.
// This behavior is different compared to 1.x version.
// If user application is exited ahead of time by calling
System.exit(N), here mark
// this application as failed with EXIT_EARLY. For a good
shutdown, user shouldn't call
// System.exit(0) to terminate the application.
finish(finalStatus,
ApplicationMaster.EXIT_EARLY,
"Shutdown hook called before final status was reported.")
}
if (!unregistered) {
// we only want to unregister if we don't want the RM to retry
if (finalStatus == FinalApplicationStatus.SUCCEEDED ||
isLastAttempt) {
unregister(finalStatus, finalMsg)
cleanupStagingDir(new
Path(System.getenv("SPARK_YARN_STAGING_DIR")))
}
}
} catch {
case e: Throwable =>
logWarning("Ignoring Exception while stopping ApplicationMaster
from shutdown hook", e)
}
}
```
`unregister` may throw exception, we should clean staging dir first.
### Why are the changes needed?
Clean staging dir
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Closes #36207 from AngersZhuuuu/SPARK-38910.
Authored-by: Angerszhuuuu <angers....@gmail.com>
Signed-off-by: Sean Owen <sro...@gmail.com>
---
.../src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
index 99ad181a542..b15623ceff5 100644
---
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
+++
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
@@ -261,8 +261,8 @@ private[spark] class ApplicationMaster(
if (!unregistered) {
// we only want to unregister if we don't want the RM to retry
if (finalStatus == FinalApplicationStatus.SUCCEEDED ||
isLastAttempt) {
- unregister(finalStatus, finalMsg)
cleanupStagingDir(new
Path(System.getenv("SPARK_YARN_STAGING_DIR")))
+ unregister(finalStatus, finalMsg)
}
}
} catch {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
