I assume your config contains "spark.yarn.credentials.file" -
otherwise startExecutorDelegationTokenRenewer(conf) call would be skipped.
On Wed, Nov 11, 2015 at 12:16 PM, Michael V Le <m...@us.ibm.com> wrote:
> Hi Ted,
>
> Thanks for reply.
>
> I tried your patch but am having the same problem.
>
> I ran:
>
> ./bin/pyspark --master yarn-client
>
> >> sc.stop()
> >> sc = SparkContext()
>
> Same error dump as below.
>
> Do I need to pass something to the new sparkcontext ?
>
> Thanks,
> Mike
>
> [image: Inactive hide details for Ted Yu ---11/11/2015 01:55:02 PM---Looks
> like the delegation token should be renewed. Mind trying the]Ted Yu
> ---11/11/2015 01:55:02 PM---Looks like the delegation token should be
> renewed. Mind trying the following ?
>
> From: Ted Yu <yuzhih...@gmail.com>
> To: Michael V Le/Watson/IBM@IBMUS
> Cc: user <user@spark.apache.org>
> Date: 11/11/2015 01:55 PM
> Subject: Re: Creating new Spark context when running in Secure YARN fails
> ------------------------------
>
>
>
> Looks like the delegation token should be renewed.
>
> Mind trying the following ?
>
> Thanks
>
> diff --git
> a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
> b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerB
> index 20771f6..e3c4a5a 100644
> ---
> a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
> +++
> b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
> @@ -53,6 +53,12 @@ private[spark] class YarnClientSchedulerBackend(
> logDebug("ClientArguments called with: " + argsArrayBuf.mkString(" "))
> val args = new ClientArguments(argsArrayBuf.toArray, conf)
> totalExpectedExecutors = args.numExecutors
> + // SPARK-8851: In yarn-client mode, the AM still does the credentials
> refresh. The driver
> + // reads the credentials from HDFS, just like the executors and
> updates its own credentials
> + // cache.
> + if (conf.contains("spark.yarn.credentials.file")) {
> + YarnSparkHadoopUtil.get.startExecutorDelegationTokenRenewer(conf)
> + }
> client = new Client(args, conf)
> appId = client.submitApplication()
>
> @@ -63,12 +69,6 @@ private[spark] class YarnClientSchedulerBackend(
>
> waitForApplication()
>
> - // SPARK-8851: In yarn-client mode, the AM still does the credentials
> refresh. The driver
> - // reads the credentials from HDFS, just like the executors and
> updates its own credentials
> - // cache.
> - if (conf.contains("spark.yarn.credentials.file")) {
> - YarnSparkHadoopUtil.get.startExecutorDelegationTokenRenewer(conf)
> - }
> monitorThread = asyncMonitorApplication()
> monitorThread.start()
> }
>
> On Wed, Nov 11, 2015 at 10:23 AM, mvle <*m...@us.ibm.com*
> <m...@us.ibm.com>> wrote:
>
> Hi,
>
> I've deployed a Secure YARN 2.7.1 cluster with HDFS encryption and am
> trying
> to run the pyspark shell using Spark 1.5.1
>
> pyspark shell works and I can run a sample code to calculate PI just
> fine.
> However, when I try to stop the current context (e.g., sc.stop()) and
> then
> create a new context (sc = SparkContext()), I get the error below.
>
> I have also seen errors such as: "token (HDFS_DELEGATION_TOKEN token
> 42 for
> hadoop) can't be found in cache",
>
> Does anyone know if it is possible to stop and create a new Spark
> context
> within a single JVM process (driver) and have that work when dealing
> with
> delegation tokens from Secure YARN/HDFS?
>
> Thanks.
>
> 15/11/11 10:19:53 INFO yarn.Client: Setting up container launch
> context for
> our AM
> 15/11/11 10:19:53 INFO yarn.Client: Setting up the launch environment
> for
> our AM container
> 15/11/11 10:19:53 INFO yarn.Client: Credentials file set to:
> credentials-37915c3e-1e90-44b9-add1-521598cea846
> 15/11/11 10:19:53 INFO yarn.YarnSparkHadoopUtil: getting token for
> namenode:
>
>
> hdfs://test6-allwkrbsec-001:9000/user/hadoop/.sparkStaging/application_1446695132208_0042
> 15/11/11 10:19:53 ERROR spark.SparkContext: Error initializing
> SparkContext.
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation
> Token
> can be issued only with kerberos or web authentication
> at
>
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:6638)
> at
>
>
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:563)
> at
>
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:987)
> at
>
>
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
>
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1476)
> at org.apache.hadoop.ipc.Client.call(Client.java:1407)
> at
>
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)
> at
>
>
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:933)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
>
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
>
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
>
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy13.getDelegationToken(Unknown Source)
> at
>
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1044)
> at
>
>
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1543)
> at
>
>
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:530)
> at
>
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508)
> at
>
>
> org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2228)
> at
>
>
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:126)
> at
>
>
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:123)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at
>
>
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokensForNamenodes(YarnSparkHadoopUtil.scala:123)
> at
>
>
> org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval(Client.scala:495)
> at
> org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:528)
> at
>
>
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:628)
> at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
> at
>
>
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
> at
>
>
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
> at
>
>
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
>
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
>
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
> at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
> at py4j.Gateway.invoke(Gateway.java:214)
> at
>
>
> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
> at
> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
> at py4j.GatewayConnection.run(GatewayConnection.java:207)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>
> --
> View this message in context:
>
> *http://apache-spark-user-list.1001560.n3.nabble.com/Creating-new-Spark-context-when-running-in-Secure-YARN-fails-tp25361.html*
>
> <http://apache-spark-user-list.1001560.n3.nabble.com/Creating-new-Spark-context-when-running-in-Secure-YARN-fails-tp25361.html>
> Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: *user-unsubscr...@spark.apache.org*
> <user-unsubscr...@spark.apache.org>
> For additional commands, e-mail: *user-h...@spark.apache.org*
> <user-h...@spark.apache.org>
>
>
>
>
