[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471259#comment-16471259
]
Yinan Li commented on SPARK-24248:
----------------------------------
Actually even if the fabric8 client does not support caching, we can
effectively achieve that and greatly simplify our code logic by doing the
following:
# Get rid of the existing in-memory data structures and replace them with a
single in-memory cache of all live executor pod objects.
# The cache is updated on every watch events. A new pod event adds one entry
to the cache, a modification event updates an existing object, and a deletion
event deletes the object.
# Always get status of an executor pod by retrieving the pod object from the
cache, falling back to talking to the API server if there's a cache miss (due
to the delay of the watch event).
Thoughts?
> [K8S] Use the Kubernetes cluster as the backing store for the state of pods
> ---------------------------------------------------------------------------
>
> Key: SPARK-24248
> URL: https://issues.apache.org/jira/browse/SPARK-24248
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 2.3.0
> Reporter: Matt Cheah
> Priority: Major
>
> We have a number of places in KubernetesClusterSchedulerBackend right now
> that maintains the state of pods in memory. However, the Kubernetes API can
> always give us the most up to date and correct view of what our executors are
> doing. We should consider moving away from in-memory state as much as can in
> favor of using the Kubernetes cluster as the source of truth for pod status.
> Maintaining less state in memory makes it so that there's a lower chance that
> we accidentally miss updating one of these data structures and breaking the
> lifecycle of executors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
