+1
On Thu, Nov 9, 2023 at 6:36 PM Xiao Li <gatorsm...@gmail.com> wrote: > > +1 > > huaxin gao <huaxin.ga...@gmail.com> 于2023年11月9日周四 16:53写道： >> >> +1 >> >> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai <dbt...@dbtsai.com> wrote: >>> >>> +1 >>> >>> To be completely transparent, I am employed in the same department as Zhou >>> at Apple. >>> >>> I support this proposal, provided that we witness community adoption >>> following the release of the Flink Kubernetes operator, streamlining Flink >>> deployment on Kubernetes. >>> >>> A well-maintained official Spark Kubernetes operator is essential for our >>> Spark community as well. >>> >>> DB Tsai | https://www.dbtsai.com/ | PGP 42E5B25A8F7A82C1 >>> >>> On Nov 9, 2023, at 12:05 PM, Zhou Jiang <zhou.c.ji...@gmail.com> wrote: >>> >>> Hi Spark community, >>> >>> I'm reaching out to initiate a conversation about the possibility of >>> developing a Java-based Kubernetes operator for Apache Spark. Following the >>> operator pattern >>> (https://kubernetes.io/docs/concepts/extend-kubernetes/operator/), Spark >>> users may manage applications and related components seamlessly using >>> native tools like kubectl. The primary goal is to simplify the Spark user >>> experience on Kubernetes, minimizing the learning curve and operational >>> complexities and therefore enable users to focus on the Spark application >>> development. >>> Although there are several open-source Spark on Kubernetes operators >>> available, none of them are officially integrated into the Apache Spark >>> project. As a result, these operators may lack active support and >>> development for new features. Within this proposal, our aim is to introduce >>> a Java-based Spark operator as an integral component of the Apache Spark >>> project. This solution has been employed internally at Apple for multiple >>> years, operating millions of executors in real production environments. The >>> use of Java in this solution is intended to accommodate a wider user and >>> contributor audience, especially those who are familiar with Scala. >>> Ideally, this operator should have its dedicated repository, similar to >>> Spark Connect Golang or Spark Docker, allowing it to maintain a loose >>> connection with the Spark release cycle. This model is also followed by the >>> Apache Flink Kubernetes operator. >>> We believe that this project holds the potential to evolve into a thriving >>> community project over the long run. A comparison can be drawn with the >>> Flink Kubernetes Operator: Apple has open-sourced internal Flink Kubernetes >>> operator, making it a part of the Apache Flink project >>> (https://github.com/apache/flink-kubernetes-operator). This move has gained >>> wide industry adoption and contributions from the community. In a mere >>> year, the Flink operator has garnered more than 600 stars and has attracted >>> contributions from over 80 contributors. This showcases the level of >>> community interest and collaborative momentum that can be achieved in >>> similar scenarios. >>> More details can be found at SPIP doc : Spark Kubernetes Operator >>> https://docs.google.com/document/d/1f5mm9VpSKeWC72Y9IiKN2jbBn32rHxjWKUfLRaGEcLE >>> >>> Thanks, >>> >>> -- >>> Zhou JIANG >>> >>> --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
