[SPARK-10812] [YARN] Fix shutdown of token renewer. A recent change to fix the referenced bug caused this exception in the `SparkContext.stop()` path:
org.apache.spark.SparkException: YarnSparkHadoopUtil is not available in non-YARN mode! at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:167) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:182) at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:440) at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1579) at org.apache.spark.SparkContext$$anonfun$stop$7.apply$mcV$sp(SparkContext.scala:1730) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1185) at org.apache.spark.SparkContext.stop(SparkContext.scala:1729) Author: Marcelo Vanzin <van...@cloudera.com> Closes #8996 from vanzin/SPARK-10812. (cherry picked from commit 4b74755122d51edb1257d4f3785fb24508681068) Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e405c2a1 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e405c2a1 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e405c2a1 Branch: refs/heads/branch-1.5 Commit: e405c2a1f6c75b50324de1bd18363b031d34f3d0 Parents: c49e0c3 Author: Marcelo Vanzin <van...@cloudera.com> Authored: Wed Oct 7 11:38:07 2015 -0700 Committer: Marcelo Vanzin <van...@cloudera.com> Committed: Thu Oct 22 13:14:26 2015 -0700 ---------------------------------------------------------------------- .../spark/scheduler/cluster/YarnClientSchedulerBackend.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/e405c2a1/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala ---------------------------------------------------------------------- diff --git a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala index d06d951..36d5759 100644 --- a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala +++ b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala @@ -178,8 +178,8 @@ private[spark] class YarnClientSchedulerBackend( monitorThread.stopMonitor() } super.stop() - client.stop() YarnSparkHadoopUtil.get.stopExecutorDelegationTokenRenewer() + client.stop() logInfo("Stopped") } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org