[ https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474774#comment-16474774 ]
Anirudh Ramanathan commented on SPARK-24248: -------------------------------------------- Somewhat related - https://issues.apache.org/jira/browse/SPARK-24266 As I understood it, there were a few in-memory data structures, but I agree with the general notion that in case there is a disconnect with the API, the driver should be resilient enough to recompute state and proceed. that should be the end goal here IMO. > [K8S] Use the Kubernetes cluster as the backing store for the state of pods > --------------------------------------------------------------------------- > > Key: SPARK-24248 > URL: https://issues.apache.org/jira/browse/SPARK-24248 > Project: Spark > Issue Type: Improvement > Components: Kubernetes > Affects Versions: 2.3.0 > Reporter: Matt Cheah > Priority: Major > > We have a number of places in KubernetesClusterSchedulerBackend right now > that maintains the state of pods in memory. However, the Kubernetes API can > always give us the most up to date and correct view of what our executors are > doing. We should consider moving away from in-memory state as much as can in > favor of using the Kubernetes cluster as the source of truth for pod status. > Maintaining less state in memory makes it so that there's a lower chance that > we accidentally miss updating one of these data structures and breaking the > lifecycle of executors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org