[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471244#comment-16471244
]
Yinan Li commented on SPARK-24248:
----------------------------------
It's potentially possible to get rid of the in-memory state in favor of getting
pod state from the pod objects directly if we are fine with the performance
penalty of communicating with the API server for each state check. One
optimization is to cache executor pod objects so retrieving them doesn't
involve network communication. This is possible with the golang client library,
but I'm not sure about the Java client we use.
> [K8S] Use the Kubernetes cluster as the backing store for the state of pods
> ---------------------------------------------------------------------------
>
> Key: SPARK-24248
> URL: https://issues.apache.org/jira/browse/SPARK-24248
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 2.3.0
> Reporter: Matt Cheah
> Priority: Major
>
> We have a number of places in KubernetesClusterSchedulerBackend right now
> that maintains the state of pods in memory. However, the Kubernetes API can
> always give us the most up to date and correct view of what our executors are
> doing. We should consider moving away from in-memory state as much as can in
> favor of using the Kubernetes cluster as the source of truth for pod status.
> Maintaining less state in memory makes it so that there's a lower chance that
> we accidentally miss updating one of these data structures and breaking the
> lifecycle of executors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
