FYI. Volcano folks are proposing more native integration with Spark. It would be great if we also get involved in the discussion.
---------- Forwarded message --------- From: Yikun Jiang <yikunk...@gmail.com> Date: Tue, Nov 30, 2021 at 12:53 AM Subject: [DISCUSSION] SPIP: Support Volcano/Alternative Schedulers Proposal To: dev <d...@spark.apache.org> Hey everyone, I'd like to start a discussion on "Support Volcano/Alternative Schedulers Proposal". This SPIP is proposed to make spark k8s schedulers provide more YARN like features (such as queues and minimum resources before scheduling jobs) that many folks want on Kubernetes. The goal of this SPIP is to improve current spark k8s scheduler implementations, add the ability of batch scheduling and support volcano as one of implementations. Design doc: https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg JIRA: https://issues.apache.org/jira/browse/SPARK-36057 Part of PRs: Ability to create resources https://github.com/apache/spark/pull/34599 Add PodGroupFeatureStep: https://github.com/apache/spark/pull/34456 Regards, Yikun