Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Further to this, I think we ought to broaden *Background and Motivation section *of the proposed doc to give more impetus to reasons behind this project. I quote " Kubernetes is

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

@Holden Karau Thanks for reminder, I will send the vote mail soon. and thanks for all helps on discussion and design review. Regards, Yikun Holden Karau 于2022年1月6日周四 03:16写道： > Do we want to move the SPIP forward to a vote? It seems like we're mostly > agreeing in principle? > > On Wed,

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 thanks Yikun for initiating this! Look forward to more progress made together by the community! On Wed, Jan 5, 2022 at 4:52 PM Weiwei Yang wrote: > +1 > > I think it is in a good shape to move forward > > On Wed, Jan 5, 2022 at 3:00 PM Bowen Li wrote: > >> +1 for SPIP >> >> According our

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 I think it is in a good shape to move forward On Wed, Jan 5, 2022 at 3:00 PM Bowen Li wrote: > +1 for SPIP > > According our production experience, the default scheduler isn't meeting > prod requirements on K8S, and such effort of integrating with batch-native > schedulers makes running

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 for SPIP According our production experience, the default scheduler isn't meeting prod requirements on K8S, and such effort of integrating with batch-native schedulers makes running Spark natively on K8S much easier for users. Thanks, Bowen On Wed, Jan 5, 2022 at 11:52 AM Mich Talebzadeh

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 non-binding view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Do we want to move the SPIP forward to a vote? It seems like we're mostly agreeing in principle? On Wed, Jan 5, 2022 at 11:12 AM Mich Talebzadeh wrote: > Hi Bo, > > Thanks for the info. Let me elaborate: > > In theory you can set the number of executors to multiple values of Nodes. > For

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi Bo, Thanks for the info. Let me elaborate: In theory you can set the number of executors to multiple values of Nodes. For example if you have a three node k8s cluster (in my case Google GKE), you can set the number of executors to 6 and end up with six executors queuing to start but

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi Mich, Curious what do you mean “The constraint seems to be that you can fit one Spark executor pod per Kubernetes node and from my tests you don't seem to be able to allocate more than 50% of RAM on the node to the container", Would you help to explain a bit? Asking this because there could be

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Thanks William for the info. The current model of Spark on k8s has certain drawbacks with pod based scheduling as I tested it on Google Kubernetes Cluster (GKE). The constraint seems to be that you can fit one Spark executor pod per Kubernetes node and from my tests you don't seem to be able to

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

; prasad.parava...@gmail.com>; John Zhuge ; Chenya Zhang > ; Chaoran Yu ; > Wilfred Spiegelenburg ; Klaus Ma < > klaus1982...@gmail.com> > *Subject:* RE: [EXTERNAL] [DISCUSSION] SPIP: Support Volcano/Alternative > Schedulers Proposal > > > > *CAUTION*: This email origin

RE: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

.platf...@gmail.com; Prasad Paravatha ; John Zhuge ; Chenya Zhang ; Chaoran Yu ; Wilfred Spiegelenburg ; Klaus Ma Subject: RE: [EXTERNAL] [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal CAUTION: This email originated from outside of the organization. Do not click links o

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Interesting,thanks Do you have any indication of the ballpark figure (a rough numerical estimate) of adding Volcano as an alternative scheduler is going to improve Spark on k8s performance? Thanks view my Linkedin profile

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi, folks! Wishing you all the best in 2022. I'd like to share the current status on "Support Customized K8S Scheduler in Spark". https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg/edit#heading=h.1quyr1r2kr5n Framework/Common support - Volcano and Yunikorn team

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

> Thank you Yikun for the info, and thanks for inviting me to a meeting to discuss this. > I appreciate your effort to put these together, and I agree that the purpose is to make Spark easy/flexible enough to support other K8s schedulers (not just for Volcano). > As discussed, could you please

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

@Yikun, Please add me to the contributors' list. Happy to help. Regards, Mich view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Thank you Yikun for the info, and thanks for inviting me to a meeting to discuss this. I appreciate your effort to put these together, and I agree that the purpose is to make Spark easy/flexible enough to support other K8s schedulers (not just for Volcano). As discussed, could you please help to

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

@Weiwei @Chenya > Thanks for bringing this up. This is quite interesting, we definitely should participate more in the discussions. Thanks for your reply and welcome to join the discussion, I think the input from Yunikorn is very critical. > The main thing here is, the Spark community should

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

IIUC, PodGroup is only supported in Volcano, this is not a common API adopted by K8s, at least not today. Spark needs to be agnostic about the schedulers, as an example, when we run Spark on YARN, does Spark need to know if that is FairScheduler or CapacityScheduler? IMO, we should build things

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi, Well, in mitigation, one cannot address all the available scheduler options at once. Certainly PodGroup is an option, unless there are reasons to believe that this is not a right choice. Therefore, I stand corrected, I fail to see where problematic comes into it, unless you may care to

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi Chenya Thanks for bringing this up. This is quite interesting, we definitely should participate more in the discussions. The main thing here is, the Spark community should make Spark pluggable in order to support other schedulers, not just for Volcano. It looks like this proposal is pushing

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Great to have other integrations and improved K8s support.  Left some comments/questions in the design doc.  TomOn Tuesday, November 30, 2021, 02:46:42 AM CST, Yikun Jiang wrote: Hey everyone, I'd like to start a discussion on "Support Volcano/Alternative Schedulers Proposal". This

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

This is a great feature/idea. I'd love to get involved in some form (testing and/or documentation). This could be my 1st contribution to Spark! On Tue, Nov 30, 2021 at 10:46 PM John Zhuge wrote: > +1 Kudos to Yikun and the community for starting the discussion! > > On Tue, Nov 30, 2021 at 8:47

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

+1 Kudos to Yikun and the community for starting the discussion! On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang wrote: > Thanks folks for bringing up the topic of natively integrating Volcano and > other alternative schedulers into Spark! > > +Weiwei, Wilfred, Chaoran. We would love to contribute

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Thanks folks for bringing up the topic of natively integrating Volcano and other alternative schedulers into Spark! +Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion as well. >From our side, we have been using and improving on one alternative resource scheduler, Apache

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Thanks for putting this together, I’m really excited for us to add better batch scheduling integrations. On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang wrote: > Hey everyone, > > I'd like to start a discussion on "Support Volcano/Alternative Schedulers > Proposal". > > This SPIP is proposed to

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hi, Good points. I have started making some suggestions to the Design doc as referenced above. My main concern with the current scheduling is the fact that there seems to be one pod per node association with k8s application. The model works on the basis of the "one-container-per-Pod" model

Re: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Adding @Holden Karau @Dongjoon Hyun @wuyi FYI On Tue, 30 Nov 2021 at 17:46, Yikun Jiang wrote: > Hey everyone, > > I'd like to start a discussion on "Support Volcano/Alternative Schedulers > Proposal". > > This SPIP is proposed to make spark k8s schedulers provide more YARN like > features

[DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal

Hey everyone, I'd like to start a discussion on "Support Volcano/Alternative Schedulers Proposal". This SPIP is proposed to make spark k8s schedulers provide more YARN like features (such as queues and minimum resources before scheduling jobs) that many folks want on Kubernetes. The goal of