[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim updated SPARK-28870: --------------------------------- Description: This issue tracks the effort on snapshotting the current status of KVStore/AppStatusListener and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add the ability to snapshot/restore of state of (SQL)AppStatusListeners. Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any compatibility, and this issue should deal with the situation: snapshot version is not compatible with SHS. was: This issue tracks the effort on snapshotting the current status of KVStore/AppStatusListener and enable incremental reading to speed up replaying event logs. This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add the ability to snapshot/restore of state of (SQL)AppStatusListeners. Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any compatibility, and this issue should deal with two situations as well: 1) some part of event log files got deleted 2) snapshot version is not compatible with SHS. > Snapshot event log files to support incremental reading > ------------------------------------------------------- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Affects Versions: 3.0.0 > Reporter: Jungtaek Lim > Priority: Major > > This issue tracks the effort on snapshotting the current status of > KVStore/AppStatusListener and enable incremental reading to speed up > replaying event logs. > This issue will be on top of SPARK-29111 and SPARK-29261, as SPARK-29111 will > add the ability to snapshot/restore from/to KVStore and SPARK-29261 will add > the ability to snapshot/restore of state of (SQL)AppStatusListeners. > Note that both SPARK-29261 and SPARK-29111 will not be guaranteeing any > compatibility, and this issue should deal with the situation: snapshot > version is not compatible with SHS. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org