GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/2924
[SPARK-3900][YARN] ApplicationMaster's shutdown hook fails and
IllegalStateException is thrown.
ApplicationMaster registers a shutdown hook and it calls
ApplicationMaster#cleanupStagingDir.
cleanupStagingDir invokes FileSystem.get(yarnConf) and it invokes
FileSystem.getInternal. FileSystem.getInternal also registers shutdown hook.
In FileSystem of hadoop 0.23, the shutdown hook registration does not
consider whether shutdown is in progress or not (In 2.2, it's considered).
// 0.23
if (map.isEmpty() ) {
ShutdownHookManager.get().addShutdownHook(clientFinalizer,
SHUTDOWN_HOOK_PRIORITY);
}
// 2.2
if (map.isEmpty()
&& !ShutdownHookManager.get().isShutdownInProgress()) {
ShutdownHookManager.get().addShutdownHook(clientFinalizer,
SHUTDOWN_HOOK_PRIORITY);
}
Thus, in 0.23, another shutdown hook can be registered when
ApplicationMaster's shutdown hook run.
This issue cause IllegalStateException as follows.
java.lang.IllegalStateException: Shutdown in progress, cannot add a
shutdownHook
at
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2306)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
at
org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$cleanupStagingDir(ApplicationMaster.scala:307)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:118)
at
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark SPARK-3900-2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2924.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2924
----
commit 57b397d27e9173b68dc3dc0d27b70b6545c9d4a0
Author: Kousuke Saruta <saru...@oss.nttdata.co.jp>
Date: 2014-10-10T16:21:28Z
Fixed IllegalStateException caused by shutdown hook registration in another
shutdown hook
commit a5f6443af483bc7c22866689252eed182607e187
Author: Kousuke Saruta <saru...@oss.nttdata.co.jp>
Date: 2014-10-12T07:30:28Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3900
commit 1cdf03c6558113d51b9e9a37b498da42bf3a25bb
Author: Kousuke Saruta <saru...@oss.nttdata.co.jp>
Date: 2014-10-14T14:35:01Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3900
commit a7d6c9bfaf8a806a74bf14cbe545baae66f8888f
Author: Kousuke Saruta <saru...@oss.nttdata.co.jp>
Date: 2014-10-14T14:35:21Z
Merge branch 'SPARK-3900' of github.com:sarutak/spark into SPARK-3900
commit ee52db2620811ba35e1a3ad1b2bc870c8213550f
Author: Kousuke Saruta <saru...@oss.nttdata.co.jp>
Date: 2014-10-15T22:07:35Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3900
commit 2c2850e37ee15c345fa6b60ee757e5bb8f51b739
Author: Kousuke Saruta <saru...@oss.nttdata.co.jp>
Date: 2014-10-17T08:34:53Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3900
commit 97018fa0e6f1cf43ce4ae25083c5a0a0a0baf35e
Author: Kousuke Saruta <saru...@oss.nttdata.co.jp>
Date: 2014-10-23T10:37:47Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3900
commit 9112817f1442fe6bb34313b686396c1886b6f2f3
Author: Kousuke Saruta <saru...@oss.nttdata.co.jp>
Date: 2014-10-24T08:33:24Z
Merge branch 'master' of git://git.apache.org/spark into SPARK-3900-2
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
