Kei Kori created SPARK-26125: -------------------------------- Summary: Delegation Token seems not appropriately stored on secrets of Kubernetes/Kerberized HDFS Key: SPARK-26125 URL: https://issues.apache.org/jira/browse/SPARK-26125 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 3.0.0 Reporter: Kei Kori Attachments: spark-submit-stern.log
I tried Kerberos authentication with Kubernetes Resource Manager and an external Hadoop and KDC. I tested built on [6c9c84f|https://github.com/apache/spark/commit/6c9c84ffb9c8d98ee2ece7ba4b010856591d383d] (master + SPARK-23257). {code} $ bin/spark-submit \ --deploy-mode cluster \ --class org.apache.spark.examples.HdfsTest \ --master k8s://https://master01.node:6443 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.app.name=spark-hdfs \ --conf spark.executer.instances=1 \ --conf spark.kubernetes.container.image=docker-registry/kkori/spark:6c9c84f \ --conf spark.kubernetes.kerberos.enabled=true \ --conf spark.kubernetes.kerberos.krb5.configMapName=krb5-conf \ --conf spark.kubernetes.kerberos.keytab=/tmp/test.keytab \ --conf spark.kubernetes.kerberos.principal=t...@external.kerberos.realm.com \ --conf spark.kubernetes.hadoop.configMapName=hadoop-conf \ local:///opt/spark/examples/jars/spark-examples_2.11-3.0.0-SNAPSHOT.jar {code} I successfully submitted into Kubernetes RM and Kubernetes spawned spark-driver and executors, but Hadoop Delegation Token seems wrongly stored into Kubernetes secrets, since that contains only header like below: {code} $ kubectl get secrets spark-hdfs-1542613661459-delegation-tokens -o jsonpath='{.data.hadoop-tokens}' | {base64 -d | cat -A; echo;} HDTS^@^@^@ {code} The result of "kubectl get secrets" should be like folloing(I masked the actual result): {code} HDTS^@^ha-hdfs:test^@^_t...@external.kerberos.realm.com^@^XXXX@ {code} As a result, spark-driver threw GSSException for each access of HDFS. Full logs(submit, driver, executor) are attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org