dongjoon-hyun opened a new pull request, #966:
URL: https://github.com/apache/spark-kubernetes-operator/pull/966
### What changes were proposed in this pull request?
This PR aims to let Kueue queue a `SparkApplication` with dynamic allocation
by reserving the quota of `spark.dynamicAllocation.maxExecutors` executors,
when the application opts in with the
`spark.operator/kueue-reserve-max-executors: "true"` annotation.
```yaml
metadata:
labels:
kueue.x-k8s.io/queue-name: spark-queue
annotations:
spark.operator/kueue-reserve-max-executors: "true"
spec:
sparkConf:
spark.dynamicAllocation.enabled: "true"
spark.dynamicAllocation.minExecutors: "1"
spark.dynamicAllocation.maxExecutors: "4"
```
- With the annotation, the executor pod set of the `Workload` requests
`spark.dynamicAllocation.maxExecutors` executors.
- It must be set, because Spark does not limit it by default.
- Without the annotation, such an application still fails with
`SchedulingFailure`, and the error message now points to the annotation.
- The `PodsReady` condition requires only the driver and
`spark.dynamicAllocation.minExecutors` (default 0) executors, since dynamic
allocation may never request the maximum. This is like the `SparkApplication`
integration of Kueue for the Kubeflow Spark Operator.
- The `Workload` records this number in its
`spark.operator/kueue-pods-ready-executor` annotation.
- The annotation is hashed together with the pod sets, so changing it
while the `Workload` is pending recreates the `Workload`, like any other spec
change.
- An application whose executors are not pods is no longer rejected for
dynamic allocation, because its `Workload` has only the driver pod set.
Examples are the `spark://` master URL of a `SparkCluster` and
`spark.kubernetes.driver.master=local[*]`.
- A `SparkCluster` with `minWorkers < maxWorkers` remains unsupported.
- The Kueue feature table now marks dynamic allocation as `Partially
supported`, and the documentation describes the drawbacks:
- The admission waits until the maximum fits.
- Preemption, fair sharing, and a `ProvisioningRequest` admission check
count the maximum for the whole attempt.
### Why are the changes needed?
Kueue cannot grow or shrink the quota of an admitted `Workload` yet. Both
[Workload slices
(KEP-77)](https://github.com/kubernetes-sigs/kueue/tree/main/keps/77-dynamically-sized-jobs)
and the in-place resize design
(https://github.com/kubernetes-sigs/kueue/pull/13646) are still evolving, so a
queued `SparkApplication` with dynamic allocation fails today.
The only workaround is to disable dynamic allocation and set
`spark.executor.instances` to the maximum. That reserves the same quota but
always runs all the executors. This opt-in stopgap reserves the same quota,
while:
- dynamic allocation still scales the executor pods, and
- the `waitForPodsReady` timeout does not evict an application which never
scales up to the maximum.
The annotation is named after its mechanism, so that the official Kueue
support can come later with its own opt-in.
### Does this PR introduce _any_ user-facing change?
No for the released versions, because the Kueue integration is not released
yet.
In the unreleased `main` branch, there are two changes:
- A `SparkApplication` with dynamic allocation can be queued with the new
annotation.
- An application whose executors are not pods is no longer rejected for
dynamic allocation.
### How was this patch tested?
Pass the CIs with the newly added test cases.
- Unit tests in `KueueWorkloadFactoryTest` and `KueueWorkloadUtilsTest`.
- A new step in the `kueue-pods-ready` E2E group runs an idle Spark Connect
server with `minExecutors=1` and `maxExecutors=2`. It checks three things:
- The server is admitted with the quota of 2 executors.
- `PodsReady` is recorded while only one executor pod exists.
- The server is not evicted after the 2-minute `waitForPodsReady` timeout
of the CI.
I also ran the `kueue-pods-ready` and `kueue` E2E groups locally on Rancher
Desktop with Kueue v0.20.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
