This is a great feature/idea. I'd love to get involved in some form (testing and/or documentation). This could be my 1st contribution to Spark!
On Tue, Nov 30, 2021 at 10:46 PM John Zhuge <jzh...@apache.org> wrote: > +1 Kudos to Yikun and the community for starting the discussion! > > On Tue, Nov 30, 2021 at 8:47 AM Chenya Zhang <chenyazhangche...@gmail.com> > wrote: > >> Thanks folks for bringing up the topic of natively integrating Volcano >> and other alternative schedulers into Spark! >> >> +Weiwei, Wilfred, Chaoran. We would love to contribute to the discussion >> as well. >> >> From our side, we have been using and improving on one alternative >> resource scheduler, Apache YuniKorn (https://yunikorn.apache.org/), for >> Spark on Kubernetes in production at Apple with solid results in the past >> year. It is capable of supporting Gang scheduling (similar to PodGroups), >> multi-tenant resource queues (similar to YARN), FIFO, and other handy >> features like bin packing to enable efficient autoscaling, etc. >> >> Natively integrating with Spark would provide more flexibility for users >> and reduce the extra cost and potential inconsistency of maintaining >> different layers of resource strategies. One interesting topic we hope to >> discuss more about is dynamic allocation, which would benefit from native >> coordination between Spark and resource schedulers in K8s & >> cloud environment for an optimal resource efficiency. >> >> >> On Tue, Nov 30, 2021 at 8:10 AM Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> Thanks for putting this together, I’m really excited for us to add >>> better batch scheduling integrations. >>> >>> On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang <yikunk...@gmail.com> >>> wrote: >>> >>>> Hey everyone, >>>> >>>> I'd like to start a discussion on "Support Volcano/Alternative >>>> Schedulers Proposal". >>>> >>>> This SPIP is proposed to make spark k8s schedulers provide more YARN >>>> like features (such as queues and minimum resources before scheduling jobs) >>>> that many folks want on Kubernetes. >>>> >>>> The goal of this SPIP is to improve current spark k8s scheduler >>>> implementations, add the ability of batch scheduling and support volcano as >>>> one of implementations. >>>> >>>> Design doc: >>>> https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg >>>> JIRA: https://issues.apache.org/jira/browse/SPARK-36057 >>>> Part of PRs: >>>> Ability to create resources https://github.com/apache/spark/pull/34599 >>>> Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456 >>>> >>>> Regards, >>>> Yikun >>>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >> > > -- > John Zhuge > -- Regards, Prasad Paravatha
