[
https://issues.apache.org/jira/browse/SPARK-58776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Florian Müller updated SPARK-58776:
-----------------------------------
Description:
{{spark.ssl.rpc.enabled=true}} does not work on Kubernetes. The keystore
passwords never reach the executors, so every executor dies while building its
{{RpcEnv}} and the application aborts without submitting a job.
{{SparkConf.isExecutorStartupConf}} withholds them deliberately ("Passwords are
propagated separately though"). That separate channel is {{{}
SecurityManager.getEnvironmentForSslRpcPasswords{}}}, which emits
* {{_SPARK_SSL_RPC_KEY_STORE_PASSWORD}}
* {{_SPARK_SSL_RPC_TRUST_STORE_PASSWORD}}
* {{_SPARK_SSL_RPC_KEY_PASSWORD}}
* {{_SPARK_SSL_RPC_PRIVATE_KEY_PASSWORD}}
read back by {{{}SSLOptions.parse{}}}. Its only caller on master is the
Standalone worker ({{{}deploy/worker/CommandUtils.scala{}}}).
{{BasicExecutorFeatureStep}} injects {{_SPARK_AUTH_SECRET}} for
{{spark.authenticate}} but never the SSL password variables, so
{{SSLOptions.parse}} sees {{None}} on the executor side.
Reproduced on {{{}apache/spark:4.1.2{}}}, Kubernetes v1.35.1, cluster mode,
PKCS#12 keystore and truststore with non-empty passphrases, passwords supplied
only in {{{}sparkConf{}}}. Executor pods receive the non-password
{{spark.ssl.rpc.}} settings as {{SPARK_JAVA_OPT_}} and none of the variables
listed above:
{code:java}
java.lang.RuntimeException: SSLFactory creation failed
at org.apache.spark.network.ssl.SSLFactory.<init>(SSLFactory.java:84)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:69)
Caused by: java.security.UnrecoverableKeyException: Get Key failed:
Cannot read the array length because "password" is null
{code}
The driver gives up after about 30 seconds on {{{}Max number of executor
failures (4) reached{}}}.
h3. Workaround
Inject the variables into the executor pods from a Secret. Only the executor
side is required, since {{SSLOptions.parse}} consults {{sparkConf}} before the
environment and the driver already resolves its passwords from the conf it was
submitted with.
{code:java}
spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_KEY_STORE_PASSWORD=<secret>:<key>
spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_KEY_PASSWORD=<secret>:<key>
spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_PRIVATE_KEY_PASSWORD=<secret>:<key>
spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_TRUST_STORE_PASSWORD=<secret>:<key>
{code}
was:
{{spark.ssl.rpc.enabled=true}} does not work on Kubernetes. The keystore
passwords never reach the executors, so every executor dies while building its
{{RpcEnv}} and the application aborts without submitting a job.
{{SparkConf.isExecutorStartupConf}} withholds them deliberately ("Passwords are
propagated separately though"). That separate channel is {{{}
SecurityManager.getEnvironmentForSslRpcPasswords{}}}, which emits
* {{_SPARK_SSL_RPC_KEY_STORE_PASSWORD}}
* {{_SPARK_SSL_RPC_TRUST_STORE_PASSWORD}}
* {{_SPARK_SSL_RPC_KEY_PASSWORD}}
* {{_SPARK_SSL_RPC_PRIVATE_KEY_PASSWORD}}
read back by {{{}SSLOptions.parse{}}}. Its only caller on master is the
Standalone worker ({{{}deploy/worker/CommandUtils.scala{}}}).
{{BasicExecutorFeatureStep}} injects {{_SPARK_AUTH_SECRET}} for
{{spark.authenticate}} but never the SSL password variables, so
{{SSLOptions.parse}} sees {{None}} on the executor side.
Reproduced on {{{}apache/spark:4.1.2{}}}, Kubernetes v1.35.1, cluster mode,
PKCS#12 keystore and truststore with non-empty passphrases, passwords supplied
only in {{{}sparkConf{}}}. Executor pods receive the non-password
{{spark.ssl.rpc.}} settings as {{SPARK_JAVA_OPT_}} and zero {{SPARK_SSL_RPC*}}
variables:
{code:java}
java.lang.RuntimeException: SSLFactory creation failed
at org.apache.spark.network.ssl.SSLFactory.<init>(SSLFactory.java:84)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:69)
Caused by: java.security.UnrecoverableKeyException: Get Key failed:
Cannot read the array length because "password" is null
{code}
The driver gives up after about 30 seconds on {{{}Max number of executor
failures (4) reached{}}}.
> K8s scheduler backend does not propagate spark.ssl.rpc.* passwords to
> executors
> -------------------------------------------------------------------------------
>
> Key: SPARK-58776
> URL: https://issues.apache.org/jira/browse/SPARK-58776
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 4.1.2
> Environment: {{{}apache/spark:4.1.2{}}}, Kubernetes v1.35.1
> Reporter: Florian Müller
> Priority: Major
> Labels: pull-request-available
>
> {{spark.ssl.rpc.enabled=true}} does not work on Kubernetes. The keystore
> passwords never reach the executors, so every executor dies while building
> its {{RpcEnv}} and the application aborts without submitting a job.
> {{SparkConf.isExecutorStartupConf}} withholds them deliberately ("Passwords
> are propagated separately though"). That separate channel is {{{}
> SecurityManager.getEnvironmentForSslRpcPasswords{}}}, which emits
> * {{_SPARK_SSL_RPC_KEY_STORE_PASSWORD}}
> * {{_SPARK_SSL_RPC_TRUST_STORE_PASSWORD}}
> * {{_SPARK_SSL_RPC_KEY_PASSWORD}}
> * {{_SPARK_SSL_RPC_PRIVATE_KEY_PASSWORD}}
> read back by {{{}SSLOptions.parse{}}}. Its only caller on master is the
> Standalone worker ({{{}deploy/worker/CommandUtils.scala{}}}).
> {{BasicExecutorFeatureStep}} injects {{_SPARK_AUTH_SECRET}} for
> {{spark.authenticate}} but never the SSL password variables, so
> {{SSLOptions.parse}} sees {{None}} on the executor side.
> Reproduced on {{{}apache/spark:4.1.2{}}}, Kubernetes v1.35.1, cluster mode,
> PKCS#12 keystore and truststore with non-empty passphrases, passwords
> supplied only in {{{}sparkConf{}}}. Executor pods receive the non-password
> {{spark.ssl.rpc.}} settings as {{SPARK_JAVA_OPT_}} and none of the variables
> listed above:
> {code:java}
> java.lang.RuntimeException: SSLFactory creation failed
> at org.apache.spark.network.ssl.SSLFactory.<init>(SSLFactory.java:84)
> at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:69)
> Caused by: java.security.UnrecoverableKeyException: Get Key failed:
> Cannot read the array length because "password" is null
> {code}
> The driver gives up after about 30 seconds on {{{}Max number of executor
> failures (4) reached{}}}.
> h3. Workaround
> Inject the variables into the executor pods from a Secret. Only the executor
> side is required, since {{SSLOptions.parse}} consults {{sparkConf}} before
> the environment and the driver already resolves its passwords from the conf
> it was submitted with.
> {code:java}
> spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_KEY_STORE_PASSWORD=<secret>:<key>
> spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_KEY_PASSWORD=<secret>:<key>
> spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_PRIVATE_KEY_PASSWORD=<secret>:<key>
> spark.kubernetes.executor.secretKeyRef._SPARK_SSL_RPC_TRUST_STORE_PASSWORD=<secret>:<key>
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
