[jira] [Commented] (SPARK-36057) Support volcano/alternative schedulers
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464959#comment-17464959 ] Apache Spark commented on SPARK-36057: -- User 'Yikun' has created a pull request for this issue: https://github.com/apache/spark/pull/35015 > Support volcano/alternative schedulers > -- > > Key: SPARK-36057 > URL: https://issues.apache.org/jira/browse/SPARK-36057 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.2.0 >Reporter: Holden Karau >Priority: Major > > This is an umbrella issue for tracking the work for supporting Volcano & > Yunikorn on Kubernetes. These schedulers provide more YARN like features > (such as queues and minimum resources before scheduling jobs) that many folks > want on Kubernetes. > > Yunikorn is an ASF project & Volcano is a CNCF project (sig-batch). > > They've taken slightly different approaches to solving the same problem, but > from Spark's point of view we should be able to share much of the code. > > See the initial brainstorming discussion in SPARK-35623. > > Google DOC: > [https://docs.google.com/document/d/1xgQGRpaHQX6-QH_J9YV2C2Dh6RpXefUpLM7KGkzL6Fg] -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36057) Support volcano/alternative schedulers
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433576#comment-17433576 ] Yikun Jiang commented on SPARK-36057: - [~seedjeffwan] Hi, jiaxin, thanks for you attention, FYI there were some discussion on [1] . The operator level change bring some k8s native dev style experience and full control of K8S resource, but there are many users want to use the native-spark style, let users can make the most of batch-scheduler by specify simple spark-submit conf. BTW, the batch scheduler have been already supported in spark-on-k8s-operator. [1] https://www.mail-archive.com/dev@spark.apache.org/msg28001.html [2] https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/tree/master/pkg/batchscheduler > Support volcano/alternative schedulers > -- > > Key: SPARK-36057 > URL: https://issues.apache.org/jira/browse/SPARK-36057 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.2.0 >Reporter: Holden Karau >Priority: Major > > This is an umbrella issue for tracking the work for supporting Volcano & > Yunikorn on Kubernetes. These schedulers provide more YARN like features > (such as queues and minimum resources before scheduling jobs) that many folks > want on Kubernetes. > > Yunikorn is an ASF project & Volcano is a CNCF project (sig-batch). > > They've taken slightly different approaches to solving the same problem, but > from Spark's point of view we should be able to share much of the code. > > See the initial brainstorming discussion in SPARK-35623. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36057) Support volcano/alternative schedulers
[ https://issues.apache.org/jira/browse/SPARK-36057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433527#comment-17433527 ] Jiaxin Shan commented on SPARK-36057: - Is it better to do this at the operator level? > Support volcano/alternative schedulers > -- > > Key: SPARK-36057 > URL: https://issues.apache.org/jira/browse/SPARK-36057 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.2.0 >Reporter: Holden Karau >Priority: Major > > This is an umbrella issue for tracking the work for supporting Volcano & > Yunikorn on Kubernetes. These schedulers provide more YARN like features > (such as queues and minimum resources before scheduling jobs) that many folks > want on Kubernetes. > > Yunikorn is an ASF project & Volcano is a CNCF project (sig-batch). > > They've taken slightly different approaches to solving the same problem, but > from Spark's point of view we should be able to share much of the code. > > See the initial brainstorming discussion in SPARK-35623. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
