[ https://issues.apache.org/jira/browse/SPARK-33485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-33485: --------------------------------- Target Version/s: (was: 3.0.0) > running spark application in kerbernetes,bug the application log shows yarn > authentications > -------------------------------------------------------------------------------------------- > > Key: SPARK-33485 > URL: https://issues.apache.org/jira/browse/SPARK-33485 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 3.0.0 > Reporter: Yuan Jiao > Priority: Major > Attachments: application.log, project.rar > > > My spark application accessing kerberized HDFS is running in kubernetes > cluster, but the application log shows: "Setting > spark.hadoop.yarn.resourcemanager.principal to tester(which is one of my > kerberos principals, yet I uses the other principal joan to read HDFS files)": > ... > + CMD=("$SPARK_HOME/bin/spark-submit" --conf > "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client > "$@") > + exec /usr/bin/tini -s – /opt/spark/bin/spark-submit --conf > spark.driver.bindAddress=10.244.1.61 --deploy-mode client --properties-file > /opt/spark/conf/spark.properties --class WordCount > local:///opt/spark/jars/WordCount-1.0-SNAPSHOT.jar > *Setting spark.hadoop.yarn.resourcemanager.principal to tester* > ... > 20/11/19 04:31:28 INFO HadoopFSDelegationTokenProvider: getting token for: > DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1041285450_1, > ugi=*tester@JOANTEST* (auth:KERBEROS)]] with renewer tester > 20/11/19 04:31:37 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 60 for > tester on ha-hdfs:nameservice1 > 20/11/19 04:31:37 INFO HadoopFSDelegationTokenProvider: getting token for: > DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1041285450_1, > ugi=*tester@JOANTEST* (auth:KERBEROS)]] with renewer tester@JOANTEST > 20/11/19 04:31:37 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 61 for > *tester* on ha-hdfs:nameservice1 > 20/11/19 04:31:37 INFO HadoopFSDelegationTokenProvider: Renewal interval is > 86400073 for token HDFS_DELEGATION_TOKEN > ... > 20/11/19 04:31:51 INFO UserGroupInformation: *Login successful for user joan > using keytab file /opt/hadoop/conf/joan.keytab* > ... > > I don't know why yarn authentication is needed here?And why use the principal > tester for autherization? Anyone can help? Thanks ! > The log and my spark project is attached blow for reference. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org