Thanks Marcelo, My understanding from a few pointers is that this may be due to insufficient read permissions to the key tab or a corrupt key tab. I have checked the read permissions and they are ok. I can see that it is initially configuring correctly:
INFO security.UserGroupInformation: Login successful for user user@login_node using keytab file /path/to/keytab I’ve added the full trace below. Gerry Full trace: Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set Spark1 will be picked by default 16/12/05 18:23:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/12/05 18:23:27 INFO security.UserGroupInformation: Login successful for user me@login_nodeusing keytab file /path/to/keytab 16/12/05 18:23:27 INFO yarn.Client: Attempting to login to the Kerberos using principal: me@login_node and keytab: /path/to/keytab 16/12/05 18:23:28 INFO impl.TimelineClientImpl: Timeline service address: http://login_node1.xcat.cluster:8188/ws/v1/timeline/ 16/12/05 18:23:28 INFO client.RMProxy: Connecting to ResourceManager at login_node1.xcat.cluster/ 16/12/05 18:23:28 INFO client.AHSProxy: Connecting to Application History server at login_node1.xcat.cluster/ 16/12/05 18:23:28 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 16/12/05 18:23:28 INFO yarn.Client: Requesting a new application from cluster with 32 NodeManagers 16/12/05 18:23:28 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (15360 MB per container) 16/12/05 18:23:28 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 16/12/05 18:23:28 INFO yarn.Client: Setting up container launch context for our AM 16/12/05 18:23:28 INFO yarn.Client: Setting up the launch environment for our AM container 16/12/05 18:23:28 INFO yarn.Client: Using the spark assembly jar on HDFS because you are using HDP, defaultSparkAssembly:hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar 16/12/05 18:23:28 INFO yarn.Client: Credentials file set to: 16/12/05 18:23:28 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_ 16/12/05 18:23:28 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1856 for me on 16/12/05 18:23:28 INFO yarn.Client: Renewal Interval set to 86400009 16/12/05 18:23:28 INFO yarn.Client: Preparing resources for our AM container 16/12/05 18:23:28 INFO yarn.YarnSparkHadoopUtil: getting token for namenode: hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_ 16/12/05 18:23:28 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1857 for me on 16/12/05 18:23:29 INFO yarn.YarnSparkHadoopUtil: HBase class not found java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration 16/12/05 18:23:29 INFO yarn.Client: To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache. 16/12/05 18:23:29 INFO yarn.Client: Uploading resource file:/path/to/keytab -> hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/keytab 16/12/05 18:23:29 INFO yarn.Client: Using the spark assembly jar on HDFS because you are using HDP, defaultSparkAssembly:hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar 16/12/05 18:23:29 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar 16/12/05 18:23:29 INFO yarn.Client: Uploading resource file:/home/me/Aoife/spark-abm/target/scala-2.10/graphx_sp_2.10-1.0.jar -> hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/graphx_sp_2.10-1.0.jar 16/12/05 18:23:29 INFO yarn.Client: Uploading resource file:/tmp/spark-2e566133-d50a-4904-920e-ab5cec07c644/__spark_conf__6538744395325375994.zip -> hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/__spark_conf__6538744395325375994.zip 16/12/05 18:23:29 INFO spark.SecurityManager: Changing view acls to: me 16/12/05 18:23:29 INFO spark.SecurityManager: Changing modify acls to: me 16/12/05 18:23:29 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(me); users with modify permissions: Set(me) 16/12/05 18:23:29 INFO yarn.Client: Submitting application 41 to ResourceManager 16/12/05 18:23:30 INFO impl.YarnClientImpl: Submitted application application_1479877553404_0041 16/12/05 18:23:31 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:31 INFO yarn.Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: AM container is launched, waiting for AM container to Register with RM ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1480962209903 final status: UNDEFINED tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/ user: me 16/12/05 18:23:32 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:33 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:34 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:35 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:36 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:37 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:38 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:39 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:40 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:40 INFO yarn.Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: N/A ApplicationMaster host: ApplicationMaster RPC port: 0 queue: default start time: 1480962209903 final status: UNDEFINED tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/ user: me 16/12/05 18:23:41 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:42 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:43 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:44 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:45 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:46 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:47 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:48 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:49 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:50 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:51 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:52 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:53 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:54 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:23:55 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:55 INFO yarn.Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: AM container is launched, waiting for AM container to Register with RM ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1480962209903 final status: UNDEFINED tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/ user: me 16/12/05 18:23:56 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:57 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:58 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:23:59 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:24:00 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:24:01 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:24:02 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:24:03 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:24:04 INFO yarn.Client: Application report for application_1479877553404_0041 (state: ACCEPTED) 16/12/05 18:24:05 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:05 INFO yarn.Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: N/A ApplicationMaster host: ApplicationMaster RPC port: 0 queue: default start time: 1480962209903 final status: UNDEFINED tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/ user: me 16/12/05 18:24:06 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:07 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:08 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:09 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:10 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:11 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:12 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:13 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:14 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:15 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:16 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:17 INFO yarn.Client: Application report for application_1479877553404_0041 (state: RUNNING) 16/12/05 18:24:18 INFO yarn.Client: Application report for application_1479877553404_0041 (state: FINISHED) 16/12/05 18:24:18 INFO yarn.Client: client token: Token { kind: YARN_CLIENT_TOKEN, service: } diagnostics: User class threw exception: org.apache.hadoop.security.AccessControlException: Authentication required ApplicationMaster host: ApplicationMaster RPC port: 0 queue: default start time: 1480962209903 final status: FAILED tracking URL: http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/ user: me Exception in thread "main" org.apache.spark.SparkException: Application application_1479877553404_0041 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1122) at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1169) at org.apache.spark.deploy.yarn.Client.main(Client.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 16/12/05 18:24:18 INFO util.ShutdownHookManager: Shutdown hook called 16/12/05 18:24:18 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2e566133-d50a-4904-920e-ab5cec07c644 On Mon, Dec 5, 2016 at 10:30 AM, Gerard Casey <gerardhughca...@gmail.com> wrote: > >> On 5 Dec 2016, at 19:26, Marcelo Vanzin <van...@cloudera.com> wrote: >> >> There's generally an exception in these cases, and you haven't posted >> it, so it's hard to tell you what's wrong. The most probable cause, >> without the extra information the exception provides, is that you're >> using the wrong Hadoop configuration when submitting the job to YARN. >> >> On Mon, Dec 5, 2016 at 4:35 AM, Gerard Casey <gerardhughca...@gmail.com> >> wrote: >>> Hello all, >>> >>> I am using Spark with Kerberos authentication. >>> >>> I can run my code using `spark-shell` fine and I can also use `spark-submit` >>> in local mode (e.g. —master local[16]). Both function as expected. >>> >>> local mode - >>> >>> spark-submit --class "graphx_sp" --master local[16] --driver-memory 20G >>> target/scala-2.10/graphx_sp_2.10-1.0.jar >>> >>> I am now progressing to run in cluster mode using YARN. >>> >>> cluster mode with YARN - >>> >>> spark-submit --class "graphx_sp" --master yarn --deploy-mode cluster >>> --executor-memory 13G --total-executor-cores 32 >>> target/scala-2.10/graphx_sp_2.10-1.0.jar >>> >>> However, this returns: >>> >>> diagnostics: User class threw exception: >>> org.apache.hadoop.security.AccessControlException: Authentication required >>> >>> Before I run using spark-shell or on local mode in spark-submit I do the >>> following kerberos setup: >>> >>> kinit -k -t ~/keytab -r 7d `whoami` >>> >>> Clearly, this setup is not extending to the YARN setup. How do I fix the >>> Kerberos issue with YARN in cluster mode? Is this something which must be in >>> my /src/main/scala/graphx_sp.scala file? >>> >>> Many thanks >>> >>> Geroid >> >> >> >> -- >> Marcelo > -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org