Thanks for putting this together, I’m really excited for us to add better batch scheduling integrations.
On Tue, Nov 30, 2021 at 12:46 AM Yikun Jiang <yikunk...@gmail.com> wrote: > Hey everyone, > > I'd like to start a discussion on "Support Volcano/Alternative Schedulers > Proposal". > > This SPIP is proposed to make spark k8s schedulers provide more YARN like > features (such as queues and minimum resources before scheduling jobs) that > many folks want on Kubernetes. > > The goal of this SPIP is to improve current spark k8s scheduler > implementations, add the ability of batch scheduling and support volcano as > one of implementations. > > Design doc: > https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg > JIRA: https://issues.apache.org/jira/browse/SPARK-36057 > Part of PRs: > Ability to create resources https://github.com/apache/spark/pull/34599 > Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456 > > Regards, > Yikun > -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
