Thanks folks for bringing up the topic of natively integrating Volcano and other alternative schedulers into Spark!
+Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion as well. >From our side, we have been using and improving on one alternative resource scheduler, Apache YuniKorn (https://yunikorn.apache.org/), for Spark on Kubernetes in production at Apple with solid results in the past year. It is capable of supporting Gang scheduling (similar to PodGroups), multi-tenant resource queues (similar to YARN), FIFO, and other handy features like bin packing to enable efficient autoscaling, etc. Natively integrating with Spark would provide more flexibility for users and reduce the extra cost and potential inconsistency of maintaining different layers of resource strategies. One interesting topic we hope to discuss more about is dynamic allocation, which would benefit from native coordination between Spark and resource schedulers in K8s & cloud environment for an optimal resource efficiency. On Tue, Nov 30, 2021 at 8:10 AM Holden Karau <hol...@pigscanfly.ca> wrote: > Thanks for putting this together, I’m really excited for us to add better > batch scheduling integrations. > > On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang <yikunk...@gmail.com> wrote: > >> Hey everyone, >> >> I'd like to start a discussion on "Support Volcano/Alternative Schedulers >> Proposal". >> >> This SPIP is proposed to make spark k8s schedulers provide more YARN like >> features (such as queues and minimum resources before scheduling jobs) that >> many folks want on Kubernetes. >> >> The goal of this SPIP is to improve current spark k8s scheduler >> implementations, add the ability of batch scheduling and support volcano as >> one of implementations. >> >> Design doc: >> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg >> JIRA: https://issues.apache.org/jira/browse/SPARK-36057 >> Part of PRs: >> Ability to create resources https://github.com/apache/spark/pull/34599 >> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456 >> >> Regards, >> Yikun >> > -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
