It looks like my config does not have "spark.yarn.credentials.file".
I executed: sc._conf.getAll() [(u'spark.ssl.keyStore', u'xxx.keystore'), (u'spark.eventLog.enabled', u'true'), (u'spark.ssl.keyStorePassword', u'XXX'), (u'spark.yarn.principal', u'XXX'), (u'spark.master', u'yarn-client'), (u'spark.ssl.keyPassword', u'XXX'), (u'spark.authenticate.sasl.serverAlwaysEncrypt', u'true'), (u'spark.ssl.trustStorePassword', u'XXX'), (u'spark.ssl.protocol', u'TLSv1.2'), (u'spark.authenticate.enableSaslEncryption', u'true'), (u'spark.app.name', u'PySparkShell'), (u'spark.yarn.keytab', u'XXX.keytab'), (u'spark.yarn.historyServer.address', u'xxx-001:18080'), (u'spark.rdd.compress', u'True'), (u'spark.eventLog.dir', u'hdfs://xxx-001:9000/user/hadoop/sparklogs'), (u'spark.ssl.enabledAlgorithms', u'TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA'), (u'spark.serializer.objectStreamReset', u'100'), (u'spark.history.fs.logDirectory', u'hdfs://xxx-001:9000/user/hadoop/sparklogs'), (u'spark.yarn.isPython', u'true'), (u'spark.submit.deployMode', u'client'), (u'spark.ssl.enabled', u'true'), (u'spark.authenticate', u'true'), (u'spark.ssl.trustStore', u'xxx.truststore')] I am not really familiar with "spark.yarn.credentials.file" and had thought it was created automatically after communicating with YARN to get tokens. Thanks, Mike From: Ted Yu <yuzhih...@gmail.com> To: Michael V Le/Watson/IBM@IBMUS Cc: user <user@spark.apache.org> Date: 11/11/2015 03:35 PM Subject: Re: Creating new Spark context when running in Secure YARN fails I assume your config contains "spark.yarn.credentials.file" - otherwise startExecutorDelegationTokenRenewer(conf) call would be skipped. On Wed, Nov 11, 2015 at 12:16 PM, Michael V Le <m...@us.ibm.com> wrote: Hi Ted, Thanks for reply. I tried your patch but am having the same problem. I ran: ./bin/pyspark --master yarn-client >> sc.stop() >> sc = SparkContext() Same error dump as below. Do I need to pass something to the new sparkcontext ? Thanks, Mike Inactive hide details for Ted Yu ---11/11/2015 01:55:02 PM---Looks like the delegation token should be renewed. Mind trying theTed Yu ---11/11/2015 01:55:02 PM---Looks like the delegation token should be renewed. Mind trying the following ? From: Ted Yu <yuzhih...@gmail.com> To: Michael V Le/Watson/IBM@IBMUS Cc: user <user@spark.apache.org> Date: 11/11/2015 01:55 PM Subject: Re: Creating new Spark context when running in Secure YARN fails Looks like the delegation token should be renewed. Mind trying the following ? Thanks diff --git a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerB index 20771f6..e3c4a5a 100644 --- a/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala +++ b/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala @@ -53,6 +53,12 @@ private[spark] class YarnClientSchedulerBackend( logDebug("ClientArguments called with: " + argsArrayBuf.mkString(" ")) val args = new ClientArguments(argsArrayBuf.toArray, conf) totalExpectedExecutors = args.numExecutors + // SPARK-8851: In yarn-client mode, the AM still does the credentials refresh. The driver + // reads the credentials from HDFS, just like the executors and updates its own credentials + // cache. + if (conf.contains("spark.yarn.credentials.file")) { + YarnSparkHadoopUtil.get.startExecutorDelegationTokenRenewer(conf) + } client = new Client(args, conf) appId = client.submitApplication() @@ -63,12 +69,6 @@ private[spark] class YarnClientSchedulerBackend( waitForApplication() - // SPARK-8851: In yarn-client mode, the AM still does the credentials refresh. The driver - // reads the credentials from HDFS, just like the executors and updates its own credentials - // cache. - if (conf.contains("spark.yarn.credentials.file")) { - YarnSparkHadoopUtil.get.startExecutorDelegationTokenRenewer(conf) - } monitorThread = asyncMonitorApplication() monitorThread.start() } On Wed, Nov 11, 2015 at 10:23 AM, mvle <m...@us.ibm.com> wrote: Hi, I've deployed a Secure YARN 2.7.1 cluster with HDFS encryption and am trying to run the pyspark shell using Spark 1.5.1 pyspark shell works and I can run a sample code to calculate PI just fine. However, when I try to stop the current context (e.g., sc.stop()) and then create a new context (sc = SparkContext()), I get the error below. I have also seen errors such as: "token (HDFS_DELEGATION_TOKEN token 42 for hadoop) can't be found in cache", Does anyone know if it is possible to stop and create a new Spark context within a single JVM process (driver) and have that work when dealing with delegation tokens from Secure YARN/HDFS? Thanks. 15/11/11 10:19:53 INFO yarn.Client: Setting up container launch context for our AM 15/11/11 10:19:53 INFO yarn.Client: Setting up the launch environment for our AM container 15/11/11 10:19:53 INFO yarn.Client: Credentials file set to: credentials-37915c3e-1e90-44b9-add1-521598cea846 15/11/11 10:19:53 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://test6-allwkrbsec-001:9000/user/hadoop/.sparkStaging/application_1446695132208_0042 15/11/11 10:19:53 ERROR spark.SparkContext: Error initializing SparkContext. org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token can be issued only with kerberos or web authentication at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken (FSNamesystem.java:6638) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken (NameNodeRpcServer.java:563) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken (ClientNamenodeProtocolServerSideTranslatorPB.java:987) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos $ClientNamenodeProtocol$2.callBlockingMethod (ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server $ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run (Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run (Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run (Server.java:2043) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1407) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke (ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken (ClientNamenodeProtocolTranslatorPB.java:933) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod (RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke (RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy13.getDelegationToken(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getDelegationToken (DFSClient.java:1044) at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken (DistributedFileSystem.java:1543) at org.apache.hadoop.fs.FileSystem.collectDelegationTokens (FileSystem.java:530) at org.apache.hadoop.fs.FileSystem.addDelegationTokens (FileSystem.java:508) at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens (DistributedFileSystem.java:2228) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun $obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:126) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun $obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:123) at scala.collection.immutable.Set$Set1.foreach (Set.scala:74) at org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokensForNamenodes (YarnSparkHadoopUtil.scala:123) at org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval (Client.scala:495) at org.apache.spark.deploy.yarn.Client.setupLaunchEnv (Client.scala:528) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext (Client.scala:628) at org.apache.spark.deploy.yarn.Client.submitApplication (Client.scala:119) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start (YarnClientSchedulerBackend.scala:56) at org.apache.spark.scheduler.TaskSchedulerImpl.start (TaskSchedulerImpl.scala:144) at org.apache.spark.SparkContext.<init> (SparkContext.scala:523) at org.apache.spark.api.java.JavaSparkContext.<init> (JavaSparkContext.scala:61) at sun.reflect.NativeConstructorAccessorImpl.newInstance0 (Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance (NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance (DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance (Constructor.java:526) at py4j.reflection.MethodInvoker.invoke (MethodInvoker.java:234) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379) at py4j.Gateway.invoke(Gateway.java:214) at py4j.commands.ConstructorCommand.invokeConstructor (ConstructorCommand.java:79) at py4j.commands.ConstructorCommand.execute (ConstructorCommand.java:68) at py4j.GatewayConnection.run(GatewayConnection.java:207) at java.lang.Thread.run(Thread.java:745) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Creating-new-Spark-context-when-running-in-Secure-YARN-fails-tp25361.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org