[ https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim updated SPARK-28870: --------------------------------- Summary: Snapshot event log files to support incremental reading (was: Snapshot old event log files to support compaction) > Snapshot event log files to support incremental reading > ------------------------------------------------------- > > Key: SPARK-28870 > URL: https://issues.apache.org/jira/browse/SPARK-28870 > Project: Spark > Issue Type: Sub-task > Components: Spark Core > Affects Versions: 3.0.0 > Reporter: Jungtaek Lim > Priority: Major > > This issue tracks the effort on compacting old event log files into snapshot > and achieve both goals, 1) reduce overall event log file size 2) speed up > replaying event logs. It also deals with cleaning event log files as snapshot > will provide the safe way to clean up old event log files without losing > ability to replay whole event logs. > This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled > event log files. This issue will be also on top of SPARK-29111 as SPARK-29111 > will add the ability to snapshot/restore from/to KVStore. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org