See SPARK-7252 If you issue query directly through hbase, does it work ?
Cheers On Tue, Dec 1, 2015 at 8:43 AM, Akhilesh Pathodia < pathodia.akhil...@gmail.com> wrote: > Spark - 1.3.1 > Hbase - 1.0.0 > Phoenix - 4.3 > Cloudera - 5.4 > > On Tue, Dec 1, 2015 at 9:35 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> What are the versions for Spark / HBase / Phoenix you're using ? >> >> Cheers >> >> On Tue, Dec 1, 2015 at 4:15 AM, Akhilesh Pathodia < >> pathodia.akhil...@gmail.com> wrote: >> >>> Hi, >>> >>> I am running spark job on yarn in cluster mode in secured cluster. Spark >>> executors are unable to get hbase connection using phoenix. I am running >>> knit command to get the ticket before starting the job and also keytab file >>> and principal are correctly specified in connection URL. But still spark >>> job on each node throws below error: >>> >>> 15/12/01 03:23:15 ERROR ipc.AbstractRpcClient: SASL authentication >>> failed. The most likely cause is missing or invalid credentials. Consider >>> 'kinit'. >>> javax.security.sasl.SaslException: GSS initiate failed [Caused by >>> GSSException: No valid credentials provided (Mechanism level: Failed to >>> find any Kerberos tgt)] >>> at >>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) >>> at >>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) >>> at >>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:605) >>> at >>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:154) >>> at >>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:731) >>> at >>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:728) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) >>> at >>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:728) >>> at >>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:881) >>> at >>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:850) >>> at >>> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1174) >>> at >>> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) >>> at >>> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) >>> at >>> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:46470) >>> at >>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1606) >>> at >>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1544) >>> at >>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1566) >>> at >>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1595) >>> at >>> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1801) >>> at >>> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) >>> at >>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) >>> at >>> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3678) >>> at >>> org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:451) >>> at >>> org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:473) >>> at >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:804) >>> at >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1194) >>> at >>> org.apache.phoenix.query.DelegateConnectionQueryServices.createTable(DelegateConnectionQueryServices.java:111) >>> at >>> org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1683) >>> at >>> org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:592) >>> at >>> org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:177) >>> at >>> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:280) >>> at >>> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:272) >>> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) >>> at >>> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:270) >>> at >>> org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1052) >>> at >>> org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1841) >>> at >>> org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1810) >>> at >>> org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77) >>> at >>> org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1810) >>> at >>> org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162) >>> at >>> org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:126) >>> at >>> org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133) >>> at >>> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) >>> at >>> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) >>> at >>> org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) >>> >>> >>> >>> I am able to connect to hbase using phoenix through a standalone >>> application in secured server which means all the configurations required >>> for getting the connection are correct. Do we need any addition >>> configuration for making phoenix work in spark job running on yarn in >>> cluster mode in secured cluster? >>> >>> How do make phoenix work with spark in secured environment? >>> >>> Thanks, >>> Akhilesh Pathodia >>> >> >> >
