Hi all,
On behalf of the Apache Spark PMC, I am proud to announce the release
of Apache Spark K8s Operator 1.1.0!
Apache Spark K8s Operator is an official subproject of Apache Spark
that manages the full lifecycle of Spark applications (SparkApplication)
and standalone Spark clusters (SparkCluster) on Kubernetes via the
Operator pattern.
Building on the 1.0.0 foundation, 1.1.0 is the first feature release
with more than 200 changes. It makes Spark a first-class citizen of
shared, multi-tenant Kubernetes clusters with native Kueue queueing,
suspend and resume, and Kubernetes Events, and it adopts Apache Spark
4.3.0 and Java 27.
Highlights of 1.1.0:
- Kueue integration: SparkApplication and SparkCluster are queued by
Kueue (v0.20.0+) with a single kueue.x-k8s.io/queue-name label.
It covers admission, Workload priority, ResourceFlavor node labels
and tolerations, admission checks such as ProvisioningRequest,
waitForPodsReady, and preemption, where a running workload is
suspended and requeued automatically. Dynamic allocation is
supported by reserving maxExecutors on opt-in. Enable it with the
operatorRbac.kueue.enabled Helm value.
- Suspend and resume: the new spec.suspend field holds a pending
SparkApplication or SparkCluster, or stops a running one into the
new Suspended state, and resumes it when unset.
- Kubernetes Events by default: Normal and Warning events for every
state transition, Kueue admission and eviction, and suspension,
visible via kubectl describe and kubectl get events.
- Apache Spark 4.3.0: the operator is built with Spark 4.3.0 and
SparkCluster uses the apache/spark:4.3.0 image by default, while
applications of Spark 4.0 and above remain supported.
- Java 27 and Kubernetes 1.37: Java 21 or newer including Java 27,
a Docker image with an Eclipse Temurin 27-based JRE on Alpine 3.24,
and Kubernetes 1.35+ recommended and verified on 1.37.
- Dynamic allocation out of the box on SparkCluster, which now
starts the external shuffle service on workers by default.
- Hardened reconciliation: honors the API server's Retry-After
hints, rides out transient API failures without failing workloads,
ignores sidecar containers of driver pods, releases the resources
of SparkClusters that fail scheduling, and fixes Prometheus
histogram metrics.
- Helm chart 1.9.0 with RBAC rules for Spark 4.2+ features, e.g.
Deployment-based executor allocation and in-place executor resize
via ExecutorResizePlugin, and sentinels in all configured
namespaces.
1.1.0 also keeps the operator the hub of the broader Apache Spark
ecosystem, with refreshed working examples for:
- Kueue, Apache YuniKorn 1.9.0, and Volcano batch schedulers
- Apache Iceberg 1.12.0 table format with Spark Connect
- Apache DataFusion Comet 1.1.0 and Apache Gluten 1.7.0 native
accelerators
- Apache Celeborn 0.7.0 remote shuffle service
- Spark 4.2+ features such as ExecutorResizePlugin,
ExecutorPVCResizePlugin, and recovery-mode executors
To get started:
helm repo add spark https://apache.github.io/spark-kubernetes-operator
helm repo update
helm install spark spark/spark-kubernetes-operator
Upgrading from 1.0.x:
Please read the migration guide before upgrading. Since helm upgrade
does not update CRDs, replace them before upgrading the operator to
use spec.suspend. Note also that the v1beta1 CRD version is deprecated
and Kubernetes Events are enabled by default.
helm repo update
helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar
kubectl replace -f spark-kubernetes-operator/crds/
-
https://github.com/apache/spark-kubernetes-operator/blob/1.1.0/docs/migration_guide.md
Release Note:
- https://github.com/apache/spark-kubernetes-operator/releases/tag/1.1.0
- https://github.com/apache/spark-kubernetes-operator/milestone/4
- https://s.apache.org/spark-kubernetes-operator-1.1.0
Published Docker Image:
- apache/spark-kubernetes-operator:1.1.0
Useful links:
- Download:
https://downloads.apache.org/spark/spark-kubernetes-operator-1.1.0/
- Website: https://s.apache.org/spark-kubernetes-operator/
- Source: https://github.com/apache/spark-kubernetes-operator
- Artifact Hub:
https://artifacthub.io/packages/search?repo=spark-kubernetes-operator
- Documentation:
https://github.com/apache/spark-kubernetes-operator/tree/main/docs
We would like to thank all the contributors, reviewers, and users who
made this release possible. This release is the result of the Apache
Spark community's collaborative effort, and we look forward to your
feedback and contributions.
Regards,
Dongjoon Hyun
on behalf of the Apache Spark PMC
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
