GitHub user skonto opened a pull request: https://github.com/apache/spark/pull/21317
[SPARK-24232][Kubernetes] Add support for secret env vars ## What changes were proposed in this pull request? * Allows to refer a secret as an env var. * Introduces new properties in the form described in the ticket. * Updates docs. * Adds required unit tests. ## How was this patch tested? Manually tested and confirmed that the secrets exist in driver's and executor's container env. First created a secret with the following yaml: ``` apiVersion: v1 kind: Secret metadata: name: test-secret data: username: c3RhdnJvcwo= password: Mzk1MjgkdmRnN0pi ------- $ echo -n 'stavros' | base64 c3RhdnJvcw== $ echo -n '1f2d1e2e67df' | base64 MWYyZDFlMmU2N2Rm ``` Run a job as follows: ```./bin/spark-submit \ --master k8s://http://localhost:9000 \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.executor.instances=1 \ --conf spark.kubernetes.container.image=skonto/spark:k8envs2 \ --conf spark.kubernetes.driver.secretKeyRef.test-secret=username \ --conf spark.kubernetes.executor.secretKeyRef.test-secret=username \ --conf spark.kubernetes.container.image.pullPolicy=Always \ local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0-SNAPSHOT.jar 10000 ``` Secret loaded correctly at the driver container: ![image](https://user-images.githubusercontent.com/7945591/39976134-b6abca52-573a-11e8-81b5-23197d3512d4.png) Also if I log into the exec container: ``` kubectl exec -it spark-pi-1526265507716-exec-1 bash bash-4.4# env SPARK_EXECUTOR_MEMORY=1g SPARK_EXECUTOR_CORES=1 LANG=C.UTF-8 HOSTNAME=spark-pi-1526265507716-exec-1 SPARK_APPLICATION_ID=spark-application-1526265518652 JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=10.100.0.1 **TEST-SECRET.USERNAME=stavros** ... ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/skonto/spark k8s-fix-env-secrets Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21317.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21317 ---- commit 24832c55e42cd768c191ba895ba054bd44d5905c Author: Stavros Kontopoulos <stavros.kontopoulos@...> Date: 2018-05-14T02:41:20Z add support for secret env vars ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org