Florian Müller created SPARK-58776:
--------------------------------------
Summary: K8s scheduler backend does not propagate spark.ssl.rpc.*
passwords to executors
Key: SPARK-58776
URL: https://issues.apache.org/jira/browse/SPARK-58776
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 4.1.2
Environment: {{{}apache/spark:4.1.2{}}}, Kubernetes v1.35.1
Reporter: Florian Müller
{{spark.ssl.rpc.enabled=true}} does not work on Kubernetes. The keystore
passwords never reach the executors, so every executor dies while building its
{{RpcEnv}} and the application aborts without submitting a job.
{{SparkConf.isExecutorStartupConf}} withholds them deliberately ("Passwords are
propagated separately though"). That separate channel is
{{{}SecurityManager.getEnvironmentForSslRpcPasswords{}}}, which emits
* {{_SPARK_SSL_RPC_KEY_STORE_PASSWORD}}
* {{_SPARK_SSL_RPC_TRUST_STORE_PASSWORD}}
* {{_SPARK_SSL_RPC_KEY_PASSWORD}}
* {{_SPARK_SSL_RPC_PRIVATE_KEY_PASSWORD}}
read back by {{{}SSLOptions.parse{}}}. Its only caller on master is the
Standalone worker
({{{}deploy/worker/CommandUtils.scala{}}}). {{BasicExecutorFeatureStep}} injects
{{_SPARK_AUTH_SECRET}} for {{spark.authenticate}} but never the SSL password
variables, so
{{SSLOptions.parse}} sees {{None}} on the executor side.
Reproduced on {{{}apache/spark:4.1.2{}}}, Kubernetes v1.35.1, cluster mode,
PKCS#12 keystore and truststore with non-empty passphrases, passwords supplied
only in {{{}sparkConf{}}}. Executor pods receive the non-password
{{spark.ssl.rpc.{*}{*}}} ** settings as {{SPARK_JAVA_OPT_}} and zero
{{SPARK_SSL_RPC*}} variables:
{code:java}
java.lang.RuntimeException: SSLFactory creation failed
at org.apache.spark.network.ssl.SSLFactory.<init>(SSLFactory.java:84)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:69)
Caused by: java.security.UnrecoverableKeyException: Get Key failed:
Cannot read the array length because "password" is null
{code}
The driver gives up after about 30 seconds on {{{}Max number of executor
failures (4) reached{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
