Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Hi Vakaris, Sorry for the late reply. Thanks for being interested in the official operator. The developers have been working on code cleaning and refactoring the internal codes for open source in the last few months. They are ready to contribute the code to Spark. We will create a dedicated

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Hi! Just wanted to inquire about the status of the official operator. We are looking forward to contributing and later on switching to a Spark Operator and we would prefer it to be the official one. Thanks, Vakaris On Thu, Nov 30, 2023 at 7:09 AM Shiqi Sun wrote: > Hi Zhou, > > Thanks for the

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Fri, Nov 10, 2023 at 8:51 PM Khalid Mammadov wrote: > +1 > > On Fri, 10 Nov 2023, 15:23 Peter Toth, wrote: > >> +1 >> >> On Fri, Nov 10, 2023, 14:09 Bjørn Jørgensen >> wrote: >> >>> +1 >>> >>> fre. 10. nov. 2023 kl. 08:39 skrev Nan Zhu : >>> just curious what happened on google’s

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Hi Zhou, Thanks for the reply. For the language choice, since I don't think I've used many k8s components written in Java on k8s, I can't really tell, but at least for the components written in Golang, they are well-organized, easy to read/maintain and run well in general. In addition, goroutines

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Hi Shiqi, Thanks for the cross-posting here - sorry for the response delay during the holiday break :) We prefer Java for the operator project as it's JVM-based and widely familiar within the Spark community. This choice aims to facilitate better adoption and ease of onboarding for future

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Hi all, Sorry for being late to the party. I went through the SPIP doc and I think this is a great proposal! I left a comment in the SPIP doc a couple days ago, but I don't see much activity there and no one replied, so I wanted to cross-post it here to get some feedback. I'm Shiqi Sun, and I

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Thanks for all the support from the community for the SPIP proposal. Since all questions/discussion are settled down (if I didn't miss any major ones), if no more questions or concerns, I'll be the shepherd for this SPIP proposal and call for a vote tomorrow. Thank you all! On Mon, Nov 13, 2023

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Hi Holden, Thanks a lot for your feedback! Yes, this proposal attempts to integrate existing solutions, especially from CRD perspective. The proposed schema retains similarity with current designs, while reducing duplicates and maintaining a single source of truth from conf properties. It also

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

To be clear: I am generally supportive of the idea (+1) but have some follow-up questions: Have we taken the time to learn from the other operators? Do we have a compatible CRD/API or not (and if so why?) The API seems to assume that everything is packaged in the container in advance, but I

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

resending cc dev for record - sorry forgot to reply all earlier :) For 1 - I'm more leaning towards 'official' as this aims to provide Spark users a community-recommended way to automate and manage Spark deployments on k8s. It does not mean the current / other options would become off-standard

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

I'd say that's actually the other way round. A user may either 1. Use spark-submit, this works with or without operator. Or, 2. Deploy the operator, create the Spark Applications with kubectl / clients - so that the Operator does spark-submit for you. We may also continue this discussion in the

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Thanks Zhou for your response to my points raised (private communication) If we start with a base model and cluster, minimal footprint for the tool, then we can establish the operational parameters needed. So +1 for me too. HTH view my Linkedin profile

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

> Not really - this is not designed to be a replacement for the current > approach. That's what I assumed too. But my question is, as a user, how to write a spark-submit command to submit a Spark app to leverage this operator? Thanks, Cheng Pan > On Nov 11, 2023, at 03:21, Zhou Jiang wrote:

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 Kazu > On Nov 10, 2023, at 10:05 AM, Khalid Mammadov > wrote: > > +1 > > On Fri, 10 Nov 2023, 15:23 Peter Toth, > wrote: >> +1 >> >> On Fri, Nov 10, 2023, 14:09 Bjørn Jørgensen > > wrote: >>> +1 >>> >>> fre. 10. nov. 2023

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Fri, 10 Nov 2023, 15:23 Peter Toth, wrote: > +1 > > On Fri, Nov 10, 2023, 14:09 Bjørn Jørgensen > wrote: > >> +1 >> >> fre. 10. nov. 2023 kl. 08:39 skrev Nan Zhu : >> >>> just curious what happened on google’s spark operator? >>> >>> On Thu, Nov 9, 2023 at 19:12 Ilan Filonenko wrote:

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Hi, Looks like a good idea but before committing myself, I have a number of design questions having looked at SPIP itself: 1. Will the name "Standard add-on Kubernetes operator to Spark '' describe it better? 2. We are still struggling with improving Spark driver start-up time.

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Fri, Nov 10, 2023, 14:09 Bjørn Jørgensen wrote: > +1 > > fre. 10. nov. 2023 kl. 08:39 skrev Nan Zhu : > >> just curious what happened on google’s spark operator? >> >> On Thu, Nov 9, 2023 at 19:12 Ilan Filonenko wrote: >> >>> +1 >>> >>> On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue wrote:

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 fre. 10. nov. 2023 kl. 08:39 skrev Nan Zhu : > just curious what happened on google’s spark operator? > > On Thu, Nov 9, 2023 at 19:12 Ilan Filonenko wrote: > >> +1 >> >> On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue wrote: >> >>> +1 >>> >>> On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala wrote:

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Fri, Nov 10, 2023 at 10:01 AM Ilan Filonenko wrote: > +1 > > On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue wrote: > >> +1 >> >> On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala wrote: >> >>> +1 for creating an official Kubernetes operator for Apache Spark >>> >>> On Fri, Nov 10, 2023 at 12:38 AM

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Thanks for this impressive proposal, I have a basic question, how does spark-submit work with this operator? Or it enforces that we must use `kubectl apply -f spark-job.yaml`(or K8s client in programming way) to submit Spark app? Thanks, Cheng Pan > On Nov 10, 2023, at 04:05, Zhou Jiang

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Thu, Nov 9, 2023 at 7:57 PM Chao Sun wrote: > > +1 > > > On Thu, Nov 9, 2023 at 6:36 PM Xiao Li wrote: > > > > +1 > > > > huaxin gao 于2023年11月9日周四 16:53写道： > >> > >> +1 > >> > >> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: > >>> > >>> +1 > >>> > >>> To be completely transparent, I am

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

just curious what happened on google’s spark operator? On Thu, Nov 9, 2023 at 19:12 Ilan Filonenko wrote: > +1 > > On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue wrote: > >> +1 >> >> On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala wrote: >> >>> +1 for creating an official Kubernetes operator for

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Thu, Nov 9, 2023 at 6:36 PM Xiao Li wrote: > > +1 > > huaxin gao 于2023年11月9日周四 16:53写道： >> >> +1 >> >> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: >>> >>> +1 >>> >>> To be completely transparent, I am employed in the same department as Zhou >>> at Apple. >>> >>> I support this

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 huaxin gao 于2023年11月9日周四 16:53写道： > +1 > > On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: > >> +1 >> >> To be completely transparent, I am employed in the same department as >> Zhou at Apple. >> >> I support this proposal, provided that we witness community adoption >> following the release

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue wrote: > +1 > > On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala wrote: > >> +1 for creating an official Kubernetes operator for Apache Spark >> >> On Fri, Nov 10, 2023 at 12:38 AM huaxin gao >> wrote: >> >>> +1 >>> >>> On Thu, Nov 9, 2023 at 3:14 PM DB

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala wrote: > +1 for creating an official Kubernetes operator for Apache Spark > > On Fri, Nov 10, 2023 at 12:38 AM huaxin gao > wrote: > >> +1 >> >> On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: >> >>> +1 >>> >>> To be completely transparent, I am

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 for creating an official Kubernetes operator for Apache Spark On Fri, Nov 10, 2023 at 12:38 AM huaxin gao wrote: > +1 > > On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: > >> +1 >> >> To be completely transparent, I am employed in the same department as >> Zhou at Apple. >> >> I support this

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Thu, Nov 9, 2023 at 3:14 PM DB Tsai wrote: > +1 > > To be completely transparent, I am employed in the same department as Zhou > at Apple. > > I support this proposal, provided that we witness community adoption > following the release of the Flink Kubernetes operator, streamlining Flink

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 To be completely transparent, I am employed in the same department as Zhou at Apple. I support this proposal, provided that we witness community adoption following the release of the Flink Kubernetes operator, streamlining Flink deployment on Kubernetes. A well-maintained official Spark

[DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Hi Spark community, I'm reaching out to initiate a conversation about the possibility of developing a Java-based Kubernetes operator for Apache Spark. Following the operator pattern ( https://kubernetes.io/docs/concepts/extend-kubernetes/operator/), Spark users may manage applications and related