Matt Cheah created SPARK-24028:
----------------------------------
Summary: [K8s] Creating secrets and config maps before creating
the driver pod has unpredictable behavior
Key: SPARK-24028
URL: https://issues.apache.org/jira/browse/SPARK-24028
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.3.0
Reporter: Matt Cheah
Currently we create the Kubernetes resources the driver depends on - such as
the properties config map and secrets to mount into the pod - only after we
create the driver pod. This is because we want these extra objects to
immediately have an owner reference to be tied to the driver pod.
On our Kubernetes 1.9.4. cluster, we're seeing that sometimes this works fine,
but other times the driver ends up being started with empty volumes instead of
volumes with the contents of the secrets we expect. The result is that
sometimes the driver will start without these files mounted, which leads to
various failures if the driver requires these files to be present early on in
their code. Missing the properties file config map, for example, would mean
spark-submit doesn't have a properties file to read at all. See the warning on
[https://kubernetes.io/docs/concepts/storage/volumes/#secret.]
Unfortunately we cannot link owner references to non-existent objects, so we
have to do this instead:
# Create the auxiliary resources without any owner references.
# Create the driver pod mounting these resources into volumes, as before.
# If #2 fails, clean up the resources created in #1.
# Edit the auxiliary resources to have an owner reference for the driver pod.
The multi-step approach leaves a small chance for us to leak resources - for
example, if we fail to make the resource edits in #4 for some reason. This also
changes the permissioning mode required for spark-submit - credentials provided
to spark-submit need to be able to edit resources in addition to creating them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
