[ https://issues.apache.org/jira/browse/SPARK-24584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-24584: ---------------------------------- Affects Version/s: (was: 2.4.0) 3.0.0 > [K8s] More efficient storage of executor pod state > -------------------------------------------------- > > Key: SPARK-24584 > URL: https://issues.apache.org/jira/browse/SPARK-24584 > Project: Spark > Issue Type: Improvement > Components: Kubernetes > Affects Versions: 3.0.0 > Reporter: Matt Cheah > Priority: Major > > Currently we store buffers of snapshots in {{ExecutorPodsSnapshotStore}}, > where the snapshots are duplicated per subscriber. With hundreds or maybe > thousands of executors, this buffering may become untenable. Investigate > storing less state while still maintaining the same level-triggered semantics. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org