HwangDongJun opened a new pull request, #58187:
URL: https://github.com/apache/spark/pull/58187
### What changes were proposed in this pull request?
When submitting a Spark application to Kubernetes in cluster mode, several
resources that the driver pod mounts as volumes were being created *after*
the driver pod itself, via the `getAdditionalKubernetesResources()` hook.
This PR switches them to use the existing (but previously unused)
`getAdditionalPreKubernetesResources()` hook instead, so they are created
*before* the driver pod:
* `KubernetesClientApplication.Client.run()`: the driver's own base config
map (mounted as `SPARK_CONF_VOLUME_DRIVER`) is now included in
`preKubernetesResources` instead of `otherKubernetesResources`.
* `HadoopConfDriverFeatureStep`: the Hadoop conf config map.
* `KerberosConfDriverFeatureStep`: the krb5 conf config map, the keytab
secret, and the delegation token secret.
* `PodTemplateConfigMapStep`: the executor pod template config map (mounted
on the driver pod, which reads it when building executor pod specs).
* `DriverKubernetesCredentialsFeatureStep`: the secret holding the driver's
submission-time Kubernetes client credentials, when
`spark.kubernetes.authenticate.driver.*` file-based credentials are used.
`MountVolumesFeatureStep` (which can create an on-demand
`PersistentVolumeClaim`)
is intentionally left unchanged: it is shared between the driver and executor
builders, and `KubernetesExecutorBuilder` has no equivalent "pre-resource"
concept, so switching it would silently break on-demand PVC creation for
executors.
### Why are the changes needed?
Kubernetes creates the driver pod and the config maps/secrets it depends on
via separate, sequential API calls. If the pod is created before its config
map/secret exists, the kubelet can attempt to mount it too early, producing
events like:
Warning FailedMount MountVolume.SetUp failed for volume
"spark-conf-volume-driver" :
configmap "spark-drv-70e59d88484c3e59-conf-map" not
found
Warning FailedMount MountVolume.SetUp failed for volume
"hadoop-properties" :
configmap
"spark-pi-6-a63eb888484c3bf1-hadoop-config" not found
The kubelet usually retries and eventually succeeds, but under slower or
more loaded API servers this race can cause the driver pod to fail to start
altogether. `KubernetesFeatureConfigStep` already provides a
`getAdditionalPreKubernetesResources()` hook meant for exactly this purpose
(resources that must exist before the pod is created), but no feature step
was using it; all of them used the post-pod-creation hook instead.
This is analogous to SPARK-38794, which fixed the same class of race for the
*executor* config map (by creating it before requesting executors). This PR
addresses the remaining occurrences on the *driver* side.
### Does this PR introduce _any_ user-facing change?
No behavior change for successful submissions. It reduces the likelihood of
transient "configmap/secret ... not found" mount warnings/failures for the
driver pod during Kubernetes cluster-mode submission.
### How was this patch tested?
* Added `ClientSuite`: `SPARK-38079: driver's own config map is created
before the driver pod, to avoid a mount race`, asserting that the first
`resourceList()` call (the pre-resources application, which happens
before the driver pod is created) includes the driver's own config map.
Reverted the fix locally and confirmed this test fails against the
original code, then confirmed it passes with the fix applied.
* Updated `ClientSuite`'s existing resource-count assertions (now the
config map is sent twice via `resourceList()`: once before pod creation,
once for the owner-reference refresh, matching the existing pattern for
other pre-resources) and added a safety check that the config map still
gets an owner reference set on the refresh call.
* Updated `HadoopConfDriverFeatureStepSuite`,
`KerberosConfDriverFeatureStepSuite`,
`PodTemplateConfigMapStepSuite`, and
`DriverKubernetesCredentialsFeatureStepSuite`
to assert against `getAdditionalPreKubernetesResources()` instead of
`getAdditionalKubernetesResources()`.
* Ran the full `kubernetes/core` module test suite: 386/386 passed.
`scalastyle`/`checkstyle`: no violations.
* Live cluster verification (Kubernetes 1.x, no custom image needed since
this is client-side submission logic): submitted `SparkPi` in cluster
mode repeatedly against the same (unmodified) driver/executor image,
toggling only which locally-built `spark-kubernetes` jar drove
`spark-submit`.
* Original code: 3 of 5 driver pods hit `FailedMount` for
`spark-conf-volume-driver` (`configmap ... not found`), matching the
reported symptom exactly. With
`spark.kubernetes.authenticate.driver.oauthToken` set to exercise
`DriverKubernetesCredentialsFeatureStep`, 2 of 5 driver pods hit
`FailedMount` for the `kubernetes-credentials` secret.
* With this patch applied: 0 `FailedMount` events across 10 runs of each
scenario; `SparkPi` completed successfully with the correct result.
* Confirmed no impact on other running services in the test namespace,
and that config maps/secrets are still garbage-collected via owner
references once the driver pod is deleted.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
