Hi Spark Team, We are facing an issue when trying to read from HDFS via spark running in standalone cluster. The issue comes from the executor node not able to authenticate. It is using auth:SIMPLE when actually we have setup auth as Kerberos. Could you please help in resolving this?
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:778) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:na] 18:57:44.726 [main] DEBUG o.a.spark.deploy.SparkHadoopUtil - creating UGI for user: <user> 18:57:45.045 [main] DEBUG o.a.h.security.UserGroupInformation - hadoop login 18:57:45.046 [main] DEBUG o.a.h.security.UserGroupInformation - hadoop login commit 18:57:45.047 [main] DEBUG o.a.h.security.UserGroupInformation - using kerberos user:<user> @GS.COM 18:57:45.047 [main] DEBUG o.a.h.security.UserGroupInformation - Using user: "<user>@GS.COM" with name <user>@GS.COM 18:57:45.047 [main] DEBUG o.a.h.security.UserGroupInformation - User entry: "<user> @GS.COM" 18:57:45.047 [main] DEBUG o.a.h.security.UserGroupInformation - UGI loginUser:<user>@GS.COM (auth:KERBEROS) 18:57:45.056 [main] DEBUG o.a.h.security.UserGroupInformation - PrivilegedAction as:<user> (auth:SIMPLE) from:org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61) 18:57:45.078 [TGT Renewer for <user>@GS.COM] DEBUG o.a.h.security.UserGroupInformation - Current time is 1674068265078 18:57:45.079 [TGT Renewer for <user>@GS.COM] DEBUG o.a.h.security.UserGroupInformation - Next refresh is 1674136785000 18:57:45.092 [main] INFO org.apache.spark.SecurityManager - Changing view acls to: root,<user> 18:57:45.092 [main] INFO org.apache.spark.SecurityManager - Changing modify acls to: root,<user> 18:57:45.093 [main] INFO org.apache.spark.SecurityManager - Changing view acls groups to: 18:57:45.093 [main] INFO org.apache.spark.SecurityManager - Changing modify acls groups to: Thanks, Jaimita Vice President, Data Lake Engineering Goldman Sachs ________________________________ Your Personal Data: We may collect and process information about you that may be subject to data protection laws. For more information about how we use and disclose your personal data, how we protect your information, our legal basis to use your information, your rights and who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>