[ https://issues.apache.org/jira/browse/SPARK-33485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuan Jiao updated SPARK-33485: ------------------------------ Description: My spark application accessing kerberized hdfs is running in kubernetes cluster, but the application log shows: "Setting spark.hadoop.yarn.resourcemanager.principal to xxx(which is my kerberos principal)": ... + SPARK_CLASSPATH='/opt/hadoop/conf::/opt/spark/jars/*' + case "$1" in + shift 1 + CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") + exec /usr/bin/tini -s – /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.1.67 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class WordCount local:///opt/spark/jars/WordCount-1.0-SNAPSHOT.jar +*Setting spark.hadoop.yarn.resourcemanager.principal to joan1*+ ... 20/11/19 05:43:07 INFO HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1695624590_1, ugi=joan1@JOANTEST (auth:KERBEROS)]] with renewer joan1 20/11/19 05:43:16 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 71 for joan1 on ha-hdfs:nameservice1 20/11/19 05:43:16 INFO HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1695624590_1, ugi=joan1@JOANTEST (auth:KERBEROS)]] with renewer joan1@JOANTEST 20/11/19 05:43:16 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 72 for joan1 on ha-hdfs:nameservice1 ... I don't know why yarn authentication is needed here? Anyone can help?Thanks! The log and my spark project is attached blow for reference. was: My spark application accessing kerberized hdfs is running in kubernetes cluster, but the application log shows: "Setting spark.hadoop.yarn.resourcemanager.principal to xxx(which is my kerberos principal)": ... + SPARK_CLASSPATH='/opt/hadoop/conf::/opt/spark/jars/*' + case "$1" in + shift 1 + CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@") + exec /usr/bin/tini -s – /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.1.67 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class WordCount local:///opt/spark/jars/WordCount-1.0-SNAPSHOT.jar +*Setting spark.hadoop.yarn.resourcemanager.principal to joan1*+ ... I don't know why yarn authentication is needed here? Anyone can help?Thanks! The log and my spark project is attached blow for reference. > running spark application in kerbernetes,bug the application log shows yarn > authentications > -------------------------------------------------------------------------------------------- > > Key: SPARK-33485 > URL: https://issues.apache.org/jira/browse/SPARK-33485 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 3.0.0 > Reporter: Yuan Jiao > Priority: Major > Attachments: application.log, project.rar > > > My spark application accessing kerberized hdfs is running in kubernetes > cluster, but the application log shows: "Setting > spark.hadoop.yarn.resourcemanager.principal to xxx(which is my kerberos > principal)": > ... > + SPARK_CLASSPATH='/opt/hadoop/conf::/opt/spark/jars/*' > + case "$1" in > + shift 1 > + CMD=("$SPARK_HOME/bin/spark-submit" --conf > "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client > "$@") > + exec /usr/bin/tini -s – /opt/spark/bin/spark-submit --conf > spark.driver.bindAddress=10.244.1.67 --deploy-mode client --properties-file > /opt/spark/conf/spark.properties --class WordCount > local:///opt/spark/jars/WordCount-1.0-SNAPSHOT.jar > +*Setting spark.hadoop.yarn.resourcemanager.principal to joan1*+ > ... > 20/11/19 05:43:07 INFO HadoopFSDelegationTokenProvider: getting token for: > DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1695624590_1, > ugi=joan1@JOANTEST (auth:KERBEROS)]] with renewer joan1 > 20/11/19 05:43:16 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 71 for > joan1 on ha-hdfs:nameservice1 > 20/11/19 05:43:16 INFO HadoopFSDelegationTokenProvider: getting token for: > DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1695624590_1, > ugi=joan1@JOANTEST (auth:KERBEROS)]] with renewer joan1@JOANTEST > 20/11/19 05:43:16 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 72 for > joan1 on ha-hdfs:nameservice1 > ... > > I don't know why yarn authentication is needed here? Anyone can help?Thanks! > The log and my spark project is attached blow for reference. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org