Zemin Piao created SPARK-51250:
----------------------------------
Summary: Add Support for K8s PriorityClass Configuration for
spark-submit via spark config
Key: SPARK-51250
URL: https://issues.apache.org/jira/browse/SPARK-51250
Project: Spark
Issue Type: New Feature
Components: Kubernetes
Affects Versions: 4.1.0
Reporter: Zemin Piao
h2. Context
Currently when running spark on k8s, with spark-submit approach, in order to
specify priority class name, the only possible way is to specify in the pod
template. In a cluster with numerous priority classes, it means multiple pod
template files are to be used, each of which has variant priority class name
only.
One could dynamically generate the pod templates based on priority classes,
however this introduces additional overhead on user side (e.g. building a
process to dynamically generate templates just for priority class name, making
sure dynamically generated templates are still valid)
Furthermore, for SPARK on YARN there is a configuration `spark.yarn.priority`
that is flexible and easy to use for users. It is good to achieve the similar
level of simplicity for users moving from SPARK on YARN to SPARK on K8s.
h2. Proposal
* Introduce a new spark configuration
{*}spark.kubernetes.priorityclass.name{*}.
* Create a new FeatureStep class that passes configured priority class name to
driver and executor pods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
