Hi, Regarding your point and I quote
".. I know that one of the Spark on Kube operators supports volcano/kube-batch so I was thinking that might be a place I would start exploring..." There seems to be ongoing work on say Volcano as part of Cloud Native Computing Foundation <https://cncf.io/> (CNCF). For example through https://github.com/volcano-sh/volcano There may be value-add in collaborating with such groups through CNCF in order to have a collective approach to such work. There also seems to be some work on Integration of Spark with Volcano for Batch Scheduling. <https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/volcano-integration.md> What is not very clear is the degree of progress of these projects. You may be kind enough to elaborate on KPI for each of these projects and where you think your contributions is going to be. HTH, Mich view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 18 Jun 2021 at 00:44, Holden Karau <hol...@pigscanfly.ca> wrote: > Hi Folks, > > I'm continuing my adventures to make Spark on containers party and I > was wondering if folks have experience with the different batch > scheduler options that they prefer? I was thinking so that we can > better support dynamic allocation it might make sense for us to > support using different schedulers and I wanted to see if there are > any that the community is more interested in? > > I know that one of the Spark on Kube operators supports > volcano/kube-batch so I was thinking that might be a place I start > exploring but also want to be open to other schedulers that folks > might be interested in. > > Cheers, > > Holden :) > > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
