zuston commented on issue #196:
URL:
https://github.com/apache/incubator-uniffle/issues/196#issuecomment-1233996276
The stacktrace is as follow
```
2022-09-01 07:31:40,131 ERROR [FlushEventThreadPool]
server.ShuffleFlushManager (ShuffleFlushManager.java:flushToFile(209)) -
Exception happened when process flush shuffle data for ShuffleDataFlushEvent:
eventId=0, appId=complexWriteTest_appId1, shuffleId=1, startPartition=0,
endPartition=1
java.lang.RuntimeException: java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Server
not found in Kerberos database (7) - Server not found in Kerberos database)];
Host Details : local host is: "fv-az489-314/10.1.1.91"; destination host is:
"localhost":37279;
at
org.apache.uniffle.storage.common.HdfsStorage.newWriteHandler(HdfsStorage.java:113)
at
org.apache.uniffle.storage.common.AbstractStorage.lambda$getOrCreateWriteHandler$2(AbstractStorage.java:50)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.uniffle.storage.common.AbstractStorage.getOrCreateWriteHandler(AbstractStorage.java:50)
at
org.apache.uniffle.server.ShuffleFlushManager.flushToFile(ShuffleFlushManager.java:168)
at
org.apache.uniffle.server.ShuffleFlushManager.lambda$null$0(ShuffleFlushManager.java:100)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: Failed on local exception:
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Server
not found in Kerberos database (7) - Server not found in Kerberos database)];
Host Details : local host is: "fv-az489-314/10.1.1.91"; destination host is:
"localhost":37279;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1493)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
at org.apache.hadoop.ipc.Client.call(Client.java:1345)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy68.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:796)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
at com.sun.proxy.$Proxy69.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1649)
at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1440)
at
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1437)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1437)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1437)
at
org.apache.uniffle.storage.handler.impl.HdfsShuffleWriteHandler.initialize(HdfsShuffleWriteHandler.java:89)
at
org.apache.uniffle.storage.handler.impl.HdfsShuffleWriteHandler.<init>(HdfsShuffleWriteHandler.java:81)
at
org.apache.uniffle.storage.common.HdfsStorage.newWriteHandler(HdfsStorage.java:108)
... 8 more
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Server not found in Kerberos database (7) - Server not found
in Kerberos database)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:755)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:718)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:811)
at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550)
at org.apache.hadoop.ipc.Client.call(Client.java:1381)
... 31 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Server not found
in Kerberos database (7) - Server not found in Kerberos database)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:406)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:410)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:798)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:794)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:793)
... 34 more
Caused by: GSSException: No valid credentials provided (Mechanism level:
Server not found in Kerberos database (7) - Server not found in Kerberos
database)
at
sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:772)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 43 more
Caused by: KrbException: Server not found in Kerberos database (7) - Server
not found in Kerberos database
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:226)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:237)
at
sun.security.krb5.internal.CredentialsUtil.serviceCredsSingle(CredentialsUtil.java:477)
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:340)
at
sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:314)
at
sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:169)
at
sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:490)
at
sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
... 46 more
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
... 54 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
