@Weiwei @Chenya
> Thanks for bringing this up. This is quite interesting, we definitely
should participate more in the discussions.
Thanks for your reply and welcome to join the discussion, I think the input
from Yunikorn is very critical.
> The main thing here is, the Spark community should
IIUC, PodGroup is only supported in Volcano, this is not a common API
adopted by K8s, at least not today.
Spark needs to be agnostic about the schedulers, as an example, when we run
Spark on YARN, does Spark need to know if that is FairScheduler or
CapacityScheduler?
IMO, we should build things
Hi,
Well, in mitigation, one cannot address all the available scheduler options
at once. Certainly PodGroup is an option, unless there are reasons to
believe that this is not a right choice. Therefore, I stand corrected, I
fail to see where problematic comes into it, unless you may care to
Hi Chenya
Thanks for bringing this up. This is quite interesting, we definitely
should participate more in the discussions.
The main thing here is, the Spark community should make Spark pluggable in
order to support other schedulers, not just for Volcano. It looks like this
proposal is pushing
Great to have other integrations and improved K8s support. Left some
comments/questions in the design doc.
TomOn Tuesday, November 30, 2021, 02:46:42 AM CST, Yikun Jiang
wrote:
Hey everyone,
I'd like to start a discussion on "Support Volcano/Alternative Schedulers
Proposal".
This
This is a great feature/idea.
I'd love to get involved in some form (testing and/or documentation). This
could be my 1st contribution to Spark!
On Tue, Nov 30, 2021 at 10:46 PM John Zhuge wrote:
> +1 Kudos to Yikun and the community for starting the discussion!
>
> On Tue, Nov 30, 2021 at 8:47
+1 Kudos to Yikun and the community for starting the discussion!
On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang
wrote:
> Thanks folks for bringing up the topic of natively integrating Volcano and
> other alternative schedulers into Spark!
>
> +Weiwei, Wilfred, Chaoran. We would love to contribute
Thanks folks for bringing up the topic of natively integrating Volcano and
other alternative schedulers into Spark!
+Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion as
well.
>From our side, we have been using and improving on one alternative resource
scheduler, Apache
Thanks for putting this together, I’m really excited for us to add better
batch scheduling integrations.
On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang wrote:
> Hey everyone,
>
> I'd like to start a discussion on "Support Volcano/Alternative Schedulers
> Proposal".
>
> This SPIP is proposed to
Hi,
Good points. I have started making some suggestions to the Design doc as
referenced above.
My main concern with the current scheduling is the fact that there seems to
be one pod per node association with k8s application. The model works on
the basis of the "one-container-per-Pod" model
Adding @Holden Karau @Dongjoon Hyun
@wuyi FYI
On Tue, 30 Nov 2021 at 17:46, Yikun Jiang wrote:
> Hey everyone,
>
> I'd like to start a discussion on "Support Volcano/Alternative Schedulers
> Proposal".
>
> This SPIP is proposed to make spark k8s schedulers provide more YARN like
> features
Hey everyone,
I'd like to start a discussion on "Support Volcano/Alternative Schedulers
Proposal".
This SPIP is proposed to make spark k8s schedulers provide more YARN like
features (such as queues and minimum resources before scheduling jobs) that
many folks want on Kubernetes.
The goal of
12 matches
Mail list logo
