Wenjun Ruan created SPARK-54825:
-----------------------------------
Summary: Supports automatic generation of Volcano podgroup minimum
resources using Spark executor/driver memory.
Key: SPARK-54825
URL: https://issues.apache.org/jira/browse/SPARK-54825
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 4.2.0
Reporter: Wenjun Ruan
Hi spark team,
I found that when using the Spark Operator to submit Spark jobs with
Volcano[1], it is not necessary to provide the full Volcano PodGroup spec. The
operator can automatically calculate the driver and executor resource requests
and set the PodGroup’s minimum resources accordingly[2]. This is a great
feature, and I hope that jobs submitted via {{spark-submit}} could also
leverage this capability.
[1]https://www.kubeflow.org/docs/components/spark-operator/user-guide/volcano-integration/
[2]https://github.com/kubeflow/spark-operator/blob/master/internal/scheduler/volcano/scheduler.go
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
