feiren kuang created KYLIN-4884:
-----------------------------------
Summary: spark cube in kerberos env
Key: KYLIN-4884
URL: https://issues.apache.org/jira/browse/KYLIN-4884
Project: Kylin
Issue Type: Bug
Components: Storage - HBase
Affects Versions: v3.1.0
Environment: OS ---> Centos 7.6
kerberos--> krb5-server-1.15.1-18
apache-kylin-3.1.0-bin-cdh60
CDH 6.3.3
HDFS 3.0.0+cdh6.3.3
YARN 3.0.0+cdh6.3.3
HBase 2.1.0+cdh6.3.3
Hive 2.1.1+cdh6.3.3
spark 2.4.0+cdh6.3.3
ZooKeeper 3.4.5+cdh6.3.3
Reporter: feiren kuang
使用spark engine
kylin start 之前 使用kinit 做好了kerberos认证 ，使用hdfs/hbase cli 访问都正常
在 8 Step Name: Convert Cuboid Data to HFile
spark log中大量 提示Kerberos认证失败；但是最终任务是成功的
21/01/23 20:43:53 INFO zookeeper.ClientCnxn: Session establishment complete on
server cdh67.test.com/192.168.1.67:2181, sessionid = 0x2768daceed1189a,
negotiated timeout = 120000
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=3, exceptions:
Sat Jan 23 20:43:56 CST 2021, RpcRetryingCaller\{globalStartTime=1611405835836,
pause=100, maxAttempts=3}, javax.security.sasl.SaslException: Call to
cdh67.test.com/192.168.1.67:16020 failed on local exception:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)] [Caused by javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]]
Sat Jan 23 20:43:56 CST 2021, RpcRetryingCaller\{globalStartTime=1611405835836,
pause=100, maxAttempts=3}, java.io.IOException: Call to
cdh67.test.com/192.168.1.67:16020 failed on local exception:
java.io.IOException: Can not send request because relogin is in progress.
Sat Jan 23 20:43:56 CST 2021, RpcRetryingCaller\{globalStartTime=1611405835836,
pause=100, maxAttempts=3}, java.io.IOException: Call to
cdh67.test.com/192.168.1.67:16020 failed on local exception:
java.io.IOException: Can not send request because relogin is in progress.
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Call to cdh67.test.com/192.168.1.67:16020
failed on local exception: java.io.IOException: Can not send request because
relogin is in progress.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:221)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:423)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:95)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:42722)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
at
org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
at
org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
at
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
... 4 more
Caused by: java.io.IOException: Can not send request because relogin is in
progress.
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection.sendRequest(NettyRpcConnection.java:301)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:421)
... 16 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
