Hi all,
On behalf of the Apache Spark PMC, I am proud to announce the release
of Apache Spark K8s Operator 1.0.0!
This is the first stable release of the Apache Spark K8s Operator, an
official subproject of Apache Spark. It marks a major milestone for the
Spark community: for the first time, the Apache Spark project itself
provides a complete, community-driven, vendor-neutral standard for
running Spark on Kubernetes via the Operator pattern.
Built on the Java Operator SDK and the Fabric8 Kubernetes client, the
operator manages the full lifecycle of Spark workloads through two
Kubernetes custom resources:
- SparkApplication: submit and manage individual Spark applications
(Java/Scala, PySpark, and SQL) with a rich state machine covering
submission, execution, failure handling, retries, and resource
cleanup.
- SparkCluster: provision and operate standalone Spark clusters,
including Spark Connect servers, Spark History Server, and
Spark Thrift Server deployments, with StatefulSet-based masters
and workers and Horizontal Pod Autoscaler support.
Highlights of 1.0.0:
- Supports Apache Spark 4.0 and above and
recent Kubernetes versions (1.34+ recommended).
- First-class Spark Connect support for next-generation
client-server Spark workloads.
- Helm-chart based installation, published on Artifact Hub, with
namespace-scoped RBAC and multi-namespace watching.
- Gateway API (HTTPRoute/GRPCRoute) integration for exposing
driver UIs and Spark Connect endpoints.
- Production-grade operations: metrics, health probes, leader
election, and extensive end-to-end test coverage via Chainsaw.
Just as importantly, 1.0.0 was designed and verified as the hub of the
broader Apache Spark ecosystem. Out of the box, it ships with working
examples and e2e coverage for:
- Apache Iceberg table format with Spark Connect
- Apache Hadoop-compatible storage and S3-compatible object stores
- Apache DataFusion Comet and Apache Gluten native accelerators
- Apache Celeborn remote shuffle service
- Apache YuniKorn and Volcano batch schedulers
We believe this release establishes the standard way to run the entire
Spark ecosystem on Kubernetes, and we are proud of what the community
has built together.
To get started:
helm repo add spark https://apache.github.io/spark-kubernetes-operator
helm repo update
helm install spark spark/spark-kubernetes-operator
Release Note:
- https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0
- https://github.com/apache/spark-kubernetes-operator/milestone/3
- https://s.apache.org/spark-kubernetes-operator-1.0.0
Published Docker Image:
- apache/spark-kubernetes-operator:1.0.0
Useful links:
- Download:
https://downloads.apache.org/spark/spark-kubernetes-operator-1.0.0/
- Website: https://s.apache.org/spark-kubernetes-operator/
- Source: https://github.com/apache/spark-kubernetes-operator
- Artifact Hub:
https://artifacthub.io/packages/search?repo=spark-kubernetes-operator
- Documentation:
https://github.com/apache/spark-kubernetes-operator/tree/main/docs
We would like to thank all the contributors, reviewers, and users who
made this first release possible. This release is the result of the
Apache Spark community's collaborative effort, and we look forward to
your feedback and contributions.
Regards,
Dongjoon Hyun
on behalf of the Apache Spark PMC
