Dongjoon Hyun created SPARK-59769:
-------------------------------------
Summary: Keep the admitted Kueue flavors when the queue label is
removed after the admission
Key: SPARK-59769
URL: https://issues.apache.org/jira/browse/SPARK-59769
Project: Spark
Issue Type: Sub-task
Components: Kubernetes
Affects Versions: kubernetes-operator-1.1.0
Reporter: Dongjoon Hyun
When the Kueue integration is enabled, removing the `kueue.x-k8s.io/queue-name`
label of a `SparkApplication` or `SparkCluster` after its `Workload` is
admitted can remove the node selector and tolerations that Kueue assigned from
its resources.
`KueueWorkloadUtils.releaseDequeuedWorkload` keeps an admitted `Workload`, but
it returns without calling `applyAdmittedFlavors`. `AppInitStep` and
`ClusterInitStep` then apply their resources again without the flavors:
- `SparkApplication`: the executor pod template ConfigMap in the driver
pre-resources is applied again. Executors created later lose the node selector
and tolerations.
- `SparkCluster`: the master and worker `StatefulSet`s are applied again with a
changed pod template, which restarts the master and worker pods.
The labeled path already handles this case. When the driver or master already
exists, it calls `applyAdmittedFlavors`.
This can happen in the following case:
1. The `Workload` is admitted, and the driver or master is created.
2. The status update to `DriverRequested` or `RunningHealthy` fails, so the
next reconciliation runs the init step again.
3. The queue label is removed before that reconciliation.
This issue exists before SPARK-59766, and it was found in the review of
https://github.com/apache/spark-kubernetes-operator/pull/881#discussion_r4096444681.
Proposed fix: apply the admitted flavors again when the `Workload` is admitted.
```java
if (workload.isEmpty()) {
return Optional.empty();
}
if (isAdmitted(workload.get())) {
return applyAdmittedFlavors(context);
}
```
Trade-off: `applyAdmittedFlavors` throws `IllegalArgumentException` on a node
selector conflict, and `ClusterInitStep` turns it into the terminal
`SchedulingFailure`. The labeled path behaves the same way, but this path would
then fail instead of changing the pod templates without notice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
