Hi Gerard, have you tried running in yarn-client mode? If so, do you still get that same error?
Regards. 2016-12-05 12:49 GMT+00:00 Gerard Casey <gerardhughca...@gmail.com>: > Edit. From here > <http://www.cloudera.com/documentation/enterprise/latest/topics/sg_spark_auth.html> > I > read that you can pass a `key tab` option to spark-submit. I thus tried > > *spark-submit --class "graphx_sp" --master yarn *--keytab > /path/to/keytab *--deploy-mode cluster --executor-memory 13G > --total-executor-cores 32 target/scala-2.10/graphx_sp_2.10-1.0.jar* > > However, the error persists > > Any ideas? > > Thanks > > Geroid > > On 5 Dec 2016, at 13:35, Gerard Casey <gerardhughca...@gmail.com> wrote: > > Hello all, > > I am using Spark with Kerberos authentication. > > I can run my code using `spark-shell` fine and I can also use > `spark-submit` in local mode (e.g. —master local[16]). Both function as > expected. > > local mode - > > *spark-submit --class "graphx_sp" --master local[16] --driver-memory 20G > target/scala-2.10/graphx_sp_2.10-1.0.jar* > > I am now progressing to run in cluster mode using YARN. > > cluster mode with YARN - > > *spark-submit --class "graphx_sp" --master yarn --deploy-mode cluster > --executor-memory 13G --total-executor-cores 32 > target/scala-2.10/graphx_sp_2.10-1.0.jar* > > However, this returns: > > *diagnostics: User class threw exception: > org.apache.hadoop.security.AccessControlException: Authentication required* > > Before I run using spark-shell or on local mode in spark-submit I do the > following kerberos setup: > > kinit -k -t ~/keytab -r 7d `whoami` > > Clearly, this setup is not extending to the YARN setup. How do I fix the > Kerberos issue with YARN in cluster mode? Is this something which must be > in my /src/main/scala/graphx_sp.scala file? > > Many thanks > > Geroid > > >
