Hi Guys, Any help regarding this issue..??
On Wed, Jan 13, 2016 at 6:39 PM, Vinay Kashyap <vinu.k...@gmail.com> wrote: > Hi all, > > I am using *Spark 1.5.1 in YARN cluster mode in CDH 5.5.* > I am trying to create an RDD by reading HBase table with kerberos enabled. > I am able to launch the spark job to read the HBase table, but I notice > that the executors launched for the job cannot proceed due to an issue with > Kerberos and they are stuck indefinitely. > > Below is my code to read a HBase table. > > > *Configuration configuration = HBaseConfiguration.create();* > * configuration.set(TableInputFormat.INPUT_TABLE, > frameStorage.getHbaseStorage().getTableId());* > * String hbaseKerberosUser = "sparkUser";* > * String hbaseKerberosKeytab = "";* > * if (!hbaseKerberosUser.trim().isEmpty() && > !hbaseKerberosKeytab.trim().isEmpty()) {* > * configuration.set("hadoop.security.authentication", "kerberos");* > * configuration.set("hbase.security.authentication", "kerberos");* > * configuration.set("hbase.security.authorization", "true");* > * configuration.set("hbase.rpc.protection", "authentication");* > * configuration.set("hbase.master.kerberos.principal", > "hbase/_HOST@CERT.LOCAL");* > * configuration.set("hbase.regionserver.kerberos.principal", > "hbase/_HOST@CERT.LOCAL");* > * configuration.set("hbase.rest.kerberos.principal", > "hbase/_HOST@CERT.LOCAL");* > * configuration.set("hbase.thrift.kerberos.principal", > "hbase/_HOST@CERT.LOCAL");* > * configuration.set("hbase.master.keytab.file", > hbaseKerberosKeytab);* > * configuration.set("hbase.regionserver.keytab.file", > hbaseKerberosKeytab);* > * configuration.set("hbase.rest.authentication.kerberos.keytab", > hbaseKerberosKeytab);* > * configuration.set("hbase.thrift.keytab.file", > hbaseKerberosKeytab);* > * UserGroupInformation.setConfiguration(configuration);* > * if (UserGroupInformation.isSecurityEnabled()) {* > * UserGroupInformation ugi = UserGroupInformation* > * .loginUserFromKeytabAndReturnUGI(hbaseKerberosUser, > hbaseKerberosKeytab);* > * TokenUtil.obtainAndCacheToken(configuration, ugi);* > * }* > * }* > > * System.out.println("loading HBase Table RDD ...");* > * JavaPairRDD<ImmutableBytesWritable, Result> hbaseTableRDD = > this.sparkContext.newAPIHadoopRDD(* > * configuration, TableInputFormat.class, > ImmutableBytesWritable.class, Result.class);* > * JavaRDD<Row> tableRDD = getTableRDD(hbaseTableRDD, dataFrameModel);* > * System.out.println("Count :: " + tableRDD.count());* > Following is the error which I can see in the container logs > > *16/01/13 10:01:42 WARN security.UserGroupInformation: > PriviledgedActionException as:sparkUser (auth:SIMPLE) > cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)]* > *16/01/13 10:01:42 WARN ipc.RpcClient: Exception encountered while > connecting to the server : javax.security.sasl.SaslException: GSS initiate > failed [Caused by GSSException: No valid credentials provided (Mechanism > level: Failed to find any Kerberos tgt)]* > *16/01/13 10:01:42 ERROR ipc.RpcClient: SASL authentication failed. The > most likely cause is missing or invalid credentials. Consider 'kinit'.* > *javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)]* > * at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)* > * at > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)* > * at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:770)* > * at > org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$600(RpcClient.java:357)* > * at > org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:891)* > * at > org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:888)* > * at java.security.AccessController.doPrivileged(Native Method)* > * at javax.security.auth.Subject.doAs(Subject.java:415)* > > Have valid Kerberos Token as can be seen below: > > sparkUser@infra:/ebs1/agent$ klist > Ticket cache: FILE:/tmp/krb5cc_1001 > Default principal: sparkUser@CERT.LOCAL > > Valid starting Expires Service principal > 13/01/2016 12:07 14/01/2016 12:07 krbtgt/CERT.LOCAL@CERT.LOCAL > > Also, I confirmed that only reading from HBase is giving this problem. > Because I can read a simple file in HDFS and I am able to create the RDD as > required. > After digging through some contents in the net, found that there is a > ticket in JIRA which is logged which is similar to what I am experiencing > *https://issues.apache.org/jira/browse/SPARK-12279 > <https://issues.apache.org/jira/browse/SPARK-12279>* > > Wanted to know if the issue is the same as I am facing..?? > And any workaround for the same so that I can proceed with my requirement > reading from HBase table.?? > > -- > *Thanks and regards* > *Vinay Kashyap* > -- *Thanks and regards* *Vinay Kashyap*