[
https://issues.apache.org/jira/browse/SPARK-46054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vara Bonthu updated SPARK-46054:
--------------------------------
Description:
*Description:*
This proposal aims to recommend the adoption of [Google's Spark K8s
Operator|https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] as the
official Spark Operator for the Apache Spark community. The operator has gained
significant traction among many users and organizations and used heavily in
production environments, but challenges related to maintenance and governance
necessitate this recommendation.
*Background:*
* Google's Spark K8s Operator is currently in use by hundreds of users and
organizations. However, due to maintenance issues, many of these users and
organizations have resorted to forking the repository and implementing their
own fixes.
* The project boasts an impressive user base with 167 contributors, 2.5k
likes, and endorsements from 45 organizations, as documented in the "Who is
using" document. Notably, there are many more organizations using it than the
initially reported 45.
* The primary issue at hand is that this project resides under the
GoogleCloudPlatform GitHub organization and is exclusively moderated by a
Google employee. Concerns have been raised by numerous users and customers
regarding the maintenance of the repository.
* The existing Google maintainers are constrained by limitations in terms of
time and support, which negatively impacts both the project and its user
community.
*Recent Developments:*
* During Kubecon Chicago 2023, AWS OSS Architects (Vara Bonthu) and the Apple
infrastructure team engaged in discussions with the Google's team, specifically
with Marcin Wielgus. They expressed their interest in contributing the project
to either the Kubeflow or Apache Spark community.
* *{color:#00875a}Marcin from Google confirmed their willingness to donate the
project to either of these communities.{color}*
* An adoption process has been initiated by the Kubeflow project under CNCF,
as documented in the following thread: [Link to the
thread|https://github.com/kubeflow/community/issues/648].
*Primary Goal:*
* The primary goal is to ensure the collaborative support and adoption of
Google's Spark Operator by the Apache Spark , thereby avoiding the development
of redundant tools and reducing confusion among users.
*Next Steps:*
* *Meeting with Apache Spark Working Group Maintainers:* We propose arranging
a meeting with the Apache Spark working group maintainers to delve deeper into
this matter, address any questions or concerns they may have, and collectively
work towards a decision.
* *Establish a New Working Group:* Upon reaching an agreement, we intend to
create a new working group comprising members from diverse organizations who
are willing to contribute and collaborate on this initiative.
* *Repository Transfer:* Our plan involves transferring the project repository
from Google's organization to either the Apache or Kubeflow organization,
aligning with the chosen community.
* *Roadmap Development:* We will formulate a new roadmap that encompasses
immediate issue resolution and a long-term design strategy aimed at enhancing
performance, scalability, and security for this tool.
We believe that working towards one Spark Operator will benefit the Apache
Spark community and address the current maintenance challenges. Your feedback
and support in this matter are highly valued. Let's collaborate to ensure a
robust and well-maintained Spark Operator for the Apache Spark community's
benefit.
*Community members are encouraged to leave their comments or give a thumbs-up
to express their support for adopting Google's Spark Operator as the official
Apache Spark operator.*
*Proposed Authors*
Vara Bonthu (AWS)
Andrey Velichkevich (Apple)
Chaoran Yu (Apple)
Marcin Wielgus (Google)
Rus Pandey (Apple)
was:
*Description:*
This proposal aims to recommend the adoption of [Google's Spark K8s
Operator|https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] as the
official Spark Operator for the Apache Spark community. The operator has gained
significant traction among many users and organizations and used heavily in
production environments, but challenges related to maintenance and governance
necessitate this recommendation.
*Background:*
* Google's Spark K8s Operator is currently in use by hundreds of users and
organizations. However, due to maintenance issues, many of these users and
organizations have resorted to forking the repository and implementing their
own fixes.
* The project boasts an impressive user base with 167 contributors, 2.5k
likes, and endorsements from 45 organizations, as documented in the "Who is
using" document. Notably, there are many more organizations using it than the
initially reported 45.
* The primary issue at hand is that this project resides under the
GoogleCloudPlatform GitHub organization and is exclusively moderated by a
Google employee. Concerns have been raised by numerous users and customers
regarding the maintenance of the repository.
* The existing Google maintainers are constrained by limitations in terms of
time and support, which negatively impacts both the project and its user
community.
*Recent Developments:*
* During Kubecon Chicago 2023, AWS OSS Architects (Vara Bonthu) and the Apple
infrastructure team engaged in discussions with the Google's team, specifically
with Marcin Wielgus. They expressed their interest in contributing the project
to either the Kubeflow or Apache Spark community.
* *{color:#00875a}Marcin from Google confirmed their willingness to donate the
project to either of these communities.{color}*
* An adoption process has been initiated by the Kubeflow project under CNCF,
as documented in the following thread: [Link to the
thread|https://github.com/kubeflow/community/issues/648].
*Primary Goal:*
**The primary goal is to ensure the collaborative support and adoption of
Google's Spark Operator by the Apache Spark (supported by Kubeflow and CNCF
communities) , thereby avoiding the development of redundant tools and reducing
confusion among users.
*Next Steps:*
* *Meeting with Apache Spark Working Group Maintainers:* We propose arranging
a meeting with the Apache Spark working group maintainers to delve deeper into
this matter, address any questions or concerns they may have, and collectively
work towards a decision.
* *Establish a New Working Group:* Upon reaching an agreement, we intend to
create a new working group comprising members from diverse organizations who
are willing to contribute and collaborate on this initiative.
* *Repository Transfer:* Our plan involves transferring the project repository
from Google's organization to either the Apache or Kubeflow organization,
aligning with the chosen community.
* *Roadmap Development:* We will formulate a new roadmap that encompasses
immediate issue resolution and a long-term design strategy aimed at enhancing
performance, scalability, and security for this tool.
We believe that working towards one Spark Operator will benefit the Apache
Spark community and address the current maintenance challenges. Your feedback
and support in this matter are highly valued. Let's collaborate to ensure a
robust and well-maintained Spark Operator for the Apache Spark community's
benefit.
*Community members are encouraged to leave their comments or give a thumbs-up
to express their support for adopting Google's Spark Operator as the official
Apache Spark operator.*
*Proposed Authors*
Vara Bonthu (AWS)
Andrey Velichkevich (Apple)
Chaoran Yu (Apple)
Marcin Wielgus (Google)
Rus Pandey (Apple)
> SPIP: Proposal to Adopt Google's Spark K8s Operator as Official Spark Operator
> ------------------------------------------------------------------------------
>
> Key: SPARK-46054
> URL: https://issues.apache.org/jira/browse/SPARK-46054
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 3.5.0
> Reporter: Vara Bonthu
> Priority: Minor
>
> *Description:*
> This proposal aims to recommend the adoption of [Google's Spark K8s
> Operator|https://github.com/GoogleCloudPlatform/spark-on-k8s-operator] as the
> official Spark Operator for the Apache Spark community. The operator has
> gained significant traction among many users and organizations and used
> heavily in production environments, but challenges related to maintenance and
> governance necessitate this recommendation.
> *Background:*
> * Google's Spark K8s Operator is currently in use by hundreds of users and
> organizations. However, due to maintenance issues, many of these users and
> organizations have resorted to forking the repository and implementing their
> own fixes.
> * The project boasts an impressive user base with 167 contributors, 2.5k
> likes, and endorsements from 45 organizations, as documented in the "Who is
> using" document. Notably, there are many more organizations using it than the
> initially reported 45.
> * The primary issue at hand is that this project resides under the
> GoogleCloudPlatform GitHub organization and is exclusively moderated by a
> Google employee. Concerns have been raised by numerous users and customers
> regarding the maintenance of the repository.
> * The existing Google maintainers are constrained by limitations in terms of
> time and support, which negatively impacts both the project and its user
> community.
>
> *Recent Developments:*
> * During Kubecon Chicago 2023, AWS OSS Architects (Vara Bonthu) and the
> Apple infrastructure team engaged in discussions with the Google's team,
> specifically with Marcin Wielgus. They expressed their interest in
> contributing the project to either the Kubeflow or Apache Spark community.
> * *{color:#00875a}Marcin from Google confirmed their willingness to donate
> the project to either of these communities.{color}*
> * An adoption process has been initiated by the Kubeflow project under CNCF,
> as documented in the following thread: [Link to the
> thread|https://github.com/kubeflow/community/issues/648].
>
> *Primary Goal:*
> * The primary goal is to ensure the collaborative support and adoption of
> Google's Spark Operator by the Apache Spark , thereby avoiding the
> development of redundant tools and reducing confusion among users.
> *Next Steps:*
> * *Meeting with Apache Spark Working Group Maintainers:* We propose
> arranging a meeting with the Apache Spark working group maintainers to delve
> deeper into this matter, address any questions or concerns they may have, and
> collectively work towards a decision.
> * *Establish a New Working Group:* Upon reaching an agreement, we intend to
> create a new working group comprising members from diverse organizations who
> are willing to contribute and collaborate on this initiative.
> * *Repository Transfer:* Our plan involves transferring the project
> repository from Google's organization to either the Apache or Kubeflow
> organization, aligning with the chosen community.
> * *Roadmap Development:* We will formulate a new roadmap that encompasses
> immediate issue resolution and a long-term design strategy aimed at enhancing
> performance, scalability, and security for this tool.
>
> We believe that working towards one Spark Operator will benefit the Apache
> Spark community and address the current maintenance challenges. Your feedback
> and support in this matter are highly valued. Let's collaborate to ensure a
> robust and well-maintained Spark Operator for the Apache Spark community's
> benefit.
> *Community members are encouraged to leave their comments or give a thumbs-up
> to express their support for adopting Google's Spark Operator as the official
> Apache Spark operator.*
>
> *Proposed Authors*
> Vara Bonthu (AWS)
> Andrey Velichkevich (Apple)
> Chaoran Yu (Apple)
> Marcin Wielgus (Google)
> Rus Pandey (Apple)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
