Fwd: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

have done some tests on Volcano with newer versions of Spark that they care to share? Thanks Forwarded Conversation Subject: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers", -- From: Mich Talebzadeh Date: Thu, 24 Feb 202

Re: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

spreadsheet with the actual size [image: 26243d69-ac3a-43f1-b2cc-903f9744237b.png] Also the spec for GKE cluster build gcloud beta container \ --project "xxx" clusters create "spark-on-gke" \ --zone "europe-west2-c" \ --no-enable-basic-auth \ --cluster-version "1.21.6-gke.1500" \

Re: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

Hi Yikun, GKE is Google's Kubernetes engine first in the market and pretty stable. The cluster deployed is a 3 node GKE with 4 Vcores and 16GB of RAM each. Autoscaling is on to take nodes from 3 to 6. So it is pretty robust. I did 15 sequences of

Re: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

@dongjoon-hyun @yangwwei Thanks! @Mich Thanks for testing it, I'm not very professional with GKE, I'm also not quite sure if it is different in configurations, internal network, scheduler implementations itself VS upstream K8S. As far as I know, different K8S vendors also maintain their own

Re: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

I did some preliminary tests without volcana and with volcano addition to spark-submit. *setup* The K8s cluster used was a Google Kubernetes standard cluster with three nodes with autoscale up to 6 nodes. It runs *spark 3.1.1* with spark-py dockers also using *spark 3.1.1 with Java 8*. In

Re: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

Hi, what do expect the performance gain to be by using volcano versus standard scheduler. Just to be sure there are two aspects here. 1. Procuring the Kubernetes cluster 2. Running the job through spark-submit Item 1 is left untouched and we should see improvements in item 2 with

Re: Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

Thank you, Yikun. I am working on SPARK-37809 and SPARK-38310 . They are the major stuff for the yunikorn part. Keep in mind we also need to add the documents. Thanks for building up the common

Recap on current status of "SPIP: Support Customized Kubernetes Schedulers"

First, much thanks for all your help (Spark/Volcano/Yunikorn community) to make this SPIP happen! Especially,@dongjoon-hyun @holdenk @william-wang @attilapiros @HyukjinKwon @martin-g @yangwwei @tgravescs The SPIP is near the end of the stage. It can be said that it is beta available at the basic