[
https://issues.apache.org/jira/browse/SPARK-46054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-46054:
---------------------------------
Fix Version/s: (was: 4.0.0)
> SPIP: Proposal to Adopt Google's Spark K8s Operator as Official Spark Operator
> ------------------------------------------------------------------------------
>
> Key: SPARK-46054
> URL: https://issues.apache.org/jira/browse/SPARK-46054
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 3.5.0
> Reporter: Vara Bonthu
> Priority: Minor
>
> *Description:*
> This proposal aims to recommend the adoption of [Google's Spark K8s
> Operator|https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] as the
> official Spark Operator for the Apache Spark community. The operator has
> gained significant traction among many users and organizations and used
> heavily in production environments, but challenges related to maintenance and
> governance necessitate this recommendation.
> *Background:*
> * Google's Spark K8s Operator is currently in use by hundreds of users and
> organizations. However, due to maintenance issues, many of these users and
> organizations have resorted to forking the repository and implementing their
> own fixes.
> * The project boasts an impressive user base with 167 contributors, 2.5k
> likes, and endorsements from 45 organizations, as documented in the "Who is
> using" document. Notably, there are many more organizations using it than the
> initially reported 45.
> * The primary issue at hand is that this project resides under the
> GoogleCloudPlatform GitHub organization and is exclusively moderated by a
> Google employee. Concerns have been raised by numerous users and customers
> regarding the maintenance of the repository.
> * The existing Google maintainers are constrained by limitations in terms of
> time and support, which negatively impacts both the project and its user
> community.
>
> *Recent Developments:*
> * During Kubecon Chicago 2023, AWS OSS Architects (Vara Bonthu) and the
> Apple infrastructure team engaged in discussions with the Google's team,
> specifically with Marcin Wielgus. They expressed their interest in
> contributing the project to either the Kubeflow or Apache Spark community.
> * *{color:#00875a}Marcin from Google confirmed their willingness to donate
> the project to either of these communities.{color}*
> * An adoption process has been initiated by the Kubeflow project under CNCF,
> as documented in the following thread: [Link to the
> thread|https://github.com/kubeflow/community/issues/648].
>
> *Primary Goal:*
> * The primary goal is to ensure the endorsement of one tool, collaboratively
> supported by the Apache Spark, Kubeflow, and CNCF communities.
>
> *Next Steps:*
> * *Meeting with Apache Spark Working Group Maintainers:* We propose
> arranging a meeting with the Apache Spark working group maintainers to delve
> deeper into this matter, address any questions or concerns they may have, and
> collectively work towards a decision.
> * *Establish a New Working Group:* Upon reaching an agreement, we intend to
> create a new working group comprising members from diverse organizations who
> are willing to contribute and collaborate on this initiative.
> * *Repository Transfer:* Our plan involves transferring the project
> repository from Google's organization to either the Apache or Kubeflow
> organization, aligning with the chosen community.
> * *Roadmap Development:* We will formulate a new roadmap that encompasses
> immediate issue resolution and a long-term design strategy aimed at enhancing
> performance, scalability, and security for this tool.
>
> We({*}Proposed Authors{*}) believe that endorsing Google's Spark K8s Operator
> as the official Spark Operator will benefit the Apache Spark community and
> address the current maintenance challenges. Your feedback and support in this
> matter are highly valued.
> Let's collaborate to ensure a robust and well-maintained Spark Operator for
> the Apache Spark community's benefit.
>
> *Community members are encouraged to leave their comments or give a thumbs-up
> to express their support for adopting Google's Spark Operator as the official
> Apache Spark operator.*
>
> *Proposed Authors*
> Vara Bonthu (AWS)
> Andrey Velichkevich (Apple)
> Chaoran Yu (Apple)
> Marcin Wielgus (Google)
> Rus Pandey (Apple)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
