Yes, its a kerberized cluster and ticket was generated using kinit command before running spark job. That's why Spark on hbase worked but when phoenix is used to get the connection to hbase, it does not pass the authentication to all nodes. Probably it is not handled in Phoenix version 4.3 or Spark 1.3.1 does not provide integration with Phoenix for kerberized cluster.
Can anybody confirm whether Spark 1.3.1 supports Phoenix on secured cluster or not? Thanks, Akhilesh On Tue, Dec 8, 2015 at 2:57 AM, Ruslan Dautkhanov <dautkha...@gmail.com> wrote: > That error is not directly related to spark nor hbase > > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)] > > Is this a kerberized cluster? You likely don't have a good (non-expired) > kerberos ticket for authentication to pass. > > > -- > Ruslan Dautkhanov > > On Mon, Dec 7, 2015 at 12:54 PM, Akhilesh Pathodia < > pathodia.akhil...@gmail.com> wrote: > >> Hi, >> >> I am running spark job on yarn in cluster mode in secured cluster. I am >> trying to run Spark on Hbase using Phoenix, but Spark executors are >> unable to get hbase connection using phoenix. I am running knit command to >> get the ticket before starting the job and also keytab file and principal >> are correctly specified in connection URL. But still spark job on each node >> throws below error: >> >> 15/12/01 03:23:15 ERROR ipc.AbstractRpcClient: SASL authentication >> failed. The most likely cause is missing or invalid credentials. Consider >> 'kinit'. >> javax.security.sasl.SaslException: GSS initiate failed [Caused by >> GSSException: No valid credentials provided (Mechanism level: Failed to >> find any Kerberos tgt)] >> at >> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) >> >> I am using Spark 1.3.1, Hbase 1.0.0, Phoenix 4.3. I am able to run Spark >> on Hbase(without phoenix) successfully in yarn-client mode as mentioned in >> this link: >> >> https://github.com/cloudera-labs/SparkOnHBase#scan-that-works-on-kerberos >> >> Also, I found that there is a known issue for yarn-cluster mode for Spark >> 1.3.1 version: >> >> https://issues.apache.org/jira/browse/SPARK-6918 >> >> Has anybody been successful in running Spark on hbase using Phoenix in >> yarn cluster or client mode? >> >> Thanks, >> Akhilesh Pathodia >> > >
