HwangDongJun opened a new pull request, #58031:
URL: https://github.com/apache/spark/pull/58031
### What changes were proposed in this pull request?
Reorders two calls in `KubernetesClusterSchedulerBackend.start()` so that
the executor
ConfigMap (`setUpExecutorConfigMap`) is created *before* executor allocation
is triggered
(`podAllocator.setTotalExpectedExecutors`), instead of after. A regression
test is added to
`KubernetesClusterSchedulerBackendSuite` asserting this ordering.
### Why are the changes needed?
`podAllocator.setTotalExpectedExecutors()` asynchronously submits executor
pod creation
requests via a background thread pool
(`ExecutorPodsSnapshotsStoreImpl.notifySubscribers()` ->
`subscribersExecutor.submit(...)`).
Previously, this was called *before* the shared executor ConfigMap was
created. This is a
genuine race: if an executor pod is scheduled and its kubelet attempts to
mount the shared
ConfigMap volume before the driver has finished creating it, the pod fails
to start with:
MountVolume.SetUp failed for volume "spark-conf-volume-exec" :
configmap "spark-exec-...-conf-map" not found
Spark's executor retry logic (and the kubelet's own mount retry) normally
recovers from this
within a few seconds, so today this surfaces only as a transient
`FailedMount` warning event
rather than a hard failure. We reproduced this repeatedly on a production
cluster during
completely normal executor startup (no fault injection needed) and captured
it via real-time
Kubernetes event logs.
This change closes the race window by giving the (synchronous) ConfigMap
creation a head
start over the async executor pod requests. It significantly narrows, but
does not perfectly
eliminate, the window, since a fully deterministic fix (e.g. blocking
executor requests until
the ConfigMap is confirmed synced to all nodes) would be a larger behavioral
change.
Note: this addresses a different failure mode than the "ConfigMap creation
permanently fails
(e.g. due to insufficient RBAC permissions) and the driver never retries"
scenario also
described in SPARK-38794. That scenario is not addressed by this change.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added `KubernetesClusterSchedulerBackendSuite` test
`SPARK-38794: executor ConfigMap is created before executors are
requested`, using
Mockito's `inOrder` to assert the ConfigMap is created before
`setTotalExpectedExecutors`
is called. Verified it fails without the fix and passes with it.
- Ran the full `kubernetes/core` test module against current master: 381/381
passing.
- Ran `./dev/scalafmt`; no reformatting was needed for the changed lines.
- Reproduced and verified the fix against a real Kubernetes cluster across
multiple driver
restarts with dynamic allocation triggering executor creation, observing
zero
`FailedMount`/`configmap not found` events with the fix applied, versus
intermittent
occurrences without it.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Sonnet 4.5 and Claude Sonnet 5 (via opencode CLI)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
