[
https://issues.apache.org/jira/browse/SPARK-38668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BoYang updated SPARK-38668:
---------------------------
Description:
Spark driver will listen to all pods events to manage its executor pods. This
will cause pressure on Kubernetes API server in a large cluster, because there
will be many drivers connect to the API server and watch for the pods.
An alternative is to have a separate service to listen and watch all pod
events. Then each Spark driver only connects to that service to get pod events.
This will reduce the load on Kubernetes API server.
> Spark on Kubernetes: support external pod watcher to reduce pressure on K8s
> API server
> --------------------------------------------------------------------------------------
>
> Key: SPARK-38668
> URL: https://issues.apache.org/jira/browse/SPARK-38668
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 3.2.1
> Reporter: BoYang
> Priority: Major
>
> Spark driver will listen to all pods events to manage its executor pods. This
> will cause pressure on Kubernetes API server in a large cluster, because
> there will be many drivers connect to the API server and watch for the pods.
>
> An alternative is to have a separate service to listen and watch all pod
> events. Then each Spark driver only connects to that service to get pod
> events. This will reduce the load on Kubernetes API server.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
