[ https://issues.apache.org/jira/browse/PHOENIX-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
MariaCarrie updated PHOENIX-5145: --------------------------------- Description: I can successfully read the data using the local mode. Here is my code: ^val sqlContext: SQLContext = missionSession.app.ss.sqlContext^ ^System.setProperty("sun.security.krb5.debug", "true")^ ^System.setProperty("sun.security.spnego.debug", "true")^ ^UserGroupInformation.loginUserFromKeytab("d...@devdip.org", "devdmp.keytab")^ ^// Load as a DataFrame directly using a Configuration object^ ^val df: DataFrame = sqlContext.phoenixTableAsDataFrame(missionSession.config.tableName, Seq("ID"), zkUrl = Some(missionSession.config.zkUrl))^ ^df.show(5)^ But when I submit this to YARN for execution, an exception will be thrown. The following is the exception information: ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:55 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:57 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:08:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:11 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:21 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:31 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:41 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:04 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:24 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)^ ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)^ ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)^ ^at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1076)^ ^at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:399)^ ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:437)^ ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:434)^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)^ ^... 55 more^ ^Caused by: java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)^ ^at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)^ ^at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)^ ^at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^ ^at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:201)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:71)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:273)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)^ ^... 1 more^ ^Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)^ ^at org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:110)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:106)^ ^at java.security.AccessController.doPrivileged(Native Method)^ ^at javax.security.auth.Subject.doAs(Subject.java:422)^ ^at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)^ ^... 18 more^ ^Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)^ ^at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)^ ^at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)^ ^at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)^ ^at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)^ ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)^ ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)^ ^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)^ ^... 30 more^ The following are my submission parameters: ^spark-submit^ ^--master yarn^ ^--name PHOENIX_SPARK_PLUGIN^ ^--deploy-mode cluster^ ^--keytab /tesdmp/keytabs/nnjKorRc37PPPjLf/dmp/testdmp.keytab^ ^--principal d...@testdip.org^ ^--jars /usr/hdp/3.0.0.0-1634/phoenix/phoenix-5.0.0.3.0.0.0-1634-client.jar^ ^--conf spark.yarn.maxAppAttempts=1^ ^/path/phoenix-spark-plugin.jar^ This program runs on YARN. Do I need to do something extra? Can anyone give me some advice? was: I can successfully read the data using the local mode. Here is my code: ^val sqlContext: SQLContext = missionSession.app.ss.sqlContext^ ^System.setProperty("sun.security.krb5.debug", "true")^ ^System.setProperty("sun.security.spnego.debug", "true")^ ^UserGroupInformation.loginUserFromKeytab("d...@devdip.org", "devdmp.keytab")^ ^// Load as a DataFrame directly using a Configuration object^ ^val df: DataFrame = sqlContext.phoenixTableAsDataFrame(missionSession.config.tableName, Seq("ID"), zkUrl = Some(missionSession.config.zkUrl))^ ^df.show(5)^ But when I submit this to YARN for execution, an exception will be thrown. The following is the exception information: ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:55 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:07:57 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^ ^Tue Feb 19 13:08:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:11 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:21 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:31 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:08:41 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:09:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:10:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:11:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:12:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:13:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:14:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:15:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:04 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:24 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^Tue Feb 19 13:16:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)^ ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)^ ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)^ ^at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1076)^ ^at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:399)^ ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:437)^ ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:434)^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)^ ^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)^ ^... 55 more^ ^Caused by: java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)^ ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)^ ^at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)^ ^at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)^ ^at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^ ^at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:201)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:71)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:273)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^ ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)^ ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)^ ^... 1 more^ ^Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^ ^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)^ ^at org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:110)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:106)^ ^at java.security.AccessController.doPrivileged(Native Method)^ ^at javax.security.auth.Subject.doAs(Subject.java:422)^ ^at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)^ ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)^ ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)^ ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)^ ^... 18 more^ ^Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)^ ^at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)^ ^at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)^ ^at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)^ ^at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)^ ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)^ ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)^ ^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)^ ^... 30 more^ The following are my submission parameters: ^spark-submit^ ^--master yarn^ ^--name PHOENIX_SPARK_PLUGIN^ ^--deploy-mode cluster^ ^--keytab /tesdmp/keytabs/nnjKorRc37PPPjLf/dmp/testdmp.keytab^ ^--principal d...@testdip.org^ ^--jars /usr/hdp/3.0.0.0-1634/phoenix/phoenix-5.0.0.3.0.0.0-1634-client.jar^ ^--conf spark.yarn.maxAppAttempts=1^ ^/opt/workspace/plugin/phoenix-spark-plugin-example-1.10.0-SNAPSHOT-jar-with-dependencies.jar "DMP_CONF=\{\"spark\":{\"sparkMaster\":\"yarn\"},\"zkUrl\":\"jdbc:phoenix:test-dmp5.fengdai.org,test-dmp3.fengdai.org,test-dmp4.fengdai.org:2181/hbase-secure\",\"tableName\":\"DMP_TEST\" ,\"isDS\":true}"^ This program runs on YARN. Do I need to do something extra? Can anyone give me some advice? > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt) > ----------------------------------------------------------------------------------------------- > > Key: PHOENIX-5145 > URL: https://issues.apache.org/jira/browse/PHOENIX-5145 > Project: Phoenix > Issue Type: Bug > Affects Versions: 5.0.0 > Environment: >HDP 3.0.0 > >Phoenix 5.0.0 > >HBase 2.0.0 > >Spark 2.3.1 > >Hadoop 3.0.1 > Reporter: MariaCarrie > Priority: Major > Original Estimate: 72h > Remaining Estimate: 72h > > I can successfully read the data using the local mode. Here is my code: > ^val sqlContext: SQLContext = missionSession.app.ss.sqlContext^ > ^System.setProperty("sun.security.krb5.debug", "true")^ > ^System.setProperty("sun.security.spnego.debug", "true")^ > ^UserGroupInformation.loginUserFromKeytab("d...@devdip.org", > "devdmp.keytab")^ > ^// Load as a DataFrame directly using a Configuration object^ > ^val df: DataFrame = > sqlContext.phoenixTableAsDataFrame(missionSession.config.tableName, > Seq("ID"), zkUrl = Some(missionSession.config.zkUrl))^ > ^df.show(5)^ > But when I submit this to YARN for execution, an exception will be thrown. > The following is the exception information: > ^Tue Feb 19 13:07:53 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: java.io.IOException: Can not send request because relogin > is in progress.^ > ^Tue Feb 19 13:07:53 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: java.io.IOException: Can not send request because relogin > is in progress.^ > ^Tue Feb 19 13:07:53 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: java.io.IOException: Can not send request because relogin > is in progress.^ > ^Tue Feb 19 13:07:54 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: java.io.IOException: Can not send request because relogin > is in progress.^ > ^Tue Feb 19 13:07:54 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: java.io.IOException: Can not send request because relogin > is in progress.^ > ^Tue Feb 19 13:07:55 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: java.io.IOException: Can not send request because relogin > is in progress.^ > ^Tue Feb 19 13:07:57 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: java.io.IOException: Can not send request because relogin > is in progress.^ > ^Tue Feb 19 13:08:01 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:08:11 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:08:21 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:08:31 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:08:41 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:09:01 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:09:22 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:09:42 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:10:02 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:10:22 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:10:42 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:11:02 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:11:22 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:11:42 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:12:02 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:12:23 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:12:43 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:13:03 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:13:23 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:13:43 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:14:03 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:14:23 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:14:43 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:15:03 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:15:23 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:15:44 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:16:04 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:16:24 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^Tue Feb 19 13:16:44 CST 2019, > RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, > java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed > on local exception: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)]^ > ^at > org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)^ > ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)^ > ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)^ > ^at > org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1076)^ > ^at > org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:399)^ > ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:437)^ > ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:434)^ > ^at > org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)^ > ^at > org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)^ > ^... 55 more^ > ^Caused by: java.io.IOException: Call to > test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)]^ > ^at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)^ > ^at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)^ > ^at > org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)^ > ^at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)^ > ^at > org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)^ > ^at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)^ > ^at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)^ > ^at > org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^ > ^at > org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)^ > ^at > org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)^ > ^at > org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)^ > ^at > org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)^ > ^at > org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:201)^ > ^at > org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:71)^ > ^at > org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:273)^ > ^at > org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)^ > ^at > org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)^ > ^... 1 more^ > ^Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused > by GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)]^ > ^at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)^ > ^at > org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)^ > ^at > org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:110)^ > ^at > org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:106)^ > ^at java.security.AccessController.doPrivileged(Native Method)^ > ^at javax.security.auth.Subject.doAs(Subject.java:422)^ > ^at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)^ > ^at > org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)^ > ^at > org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)^ > ^at > org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)^ > ^... 18 more^ > ^Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)^ > ^at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)^ > ^at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)^ > ^at > sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)^ > ^at > sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)^ > ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)^ > ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)^ > ^at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)^ > ^... 30 more^ > The following are my submission parameters: > ^spark-submit^ > ^--master yarn^ > ^--name PHOENIX_SPARK_PLUGIN^ > ^--deploy-mode cluster^ > ^--keytab /tesdmp/keytabs/nnjKorRc37PPPjLf/dmp/testdmp.keytab^ > ^--principal d...@testdip.org^ > ^--jars /usr/hdp/3.0.0.0-1634/phoenix/phoenix-5.0.0.3.0.0.0-1634-client.jar^ > > ^--conf spark.yarn.maxAppAttempts=1^ > ^/path/phoenix-spark-plugin.jar^ > This program runs on YARN. Do I need to do something extra? Can anyone give > me some advice? > -- This message was sent by Atlassian JIRA (v7.6.3#76005)