[
https://issues.apache.org/jira/browse/SPARK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-28870:
---------------------------------
Description:
This issue tracks the effort on compacting event log files into snapshot and
enable incremental reading to speed up replaying event logs.
This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled
event log files. This issue will be also on top of SPARK-29111 as SPARK-29111
will add the ability to snapshot/restore from/to KVStore.
was:
This issue tracks the effort on compacting old event log files into snapshot
and achieve both goals, 1) reduce overall event log file size 2) speed up
replaying event logs. It also deals with cleaning event log files as snapshot
will provide the safe way to clean up old event log files without losing
ability to replay whole event logs.
This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled
event log files. This issue will be also on top of SPARK-29111 as SPARK-29111
will add the ability to snapshot/restore from/to KVStore.
> Snapshot event log files to support incremental reading
> -------------------------------------------------------
>
> Key: SPARK-28870
> URL: https://issues.apache.org/jira/browse/SPARK-28870
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> This issue tracks the effort on compacting event log files into snapshot and
> enable incremental reading to speed up replaying event logs.
> This issue will be on top of SPARK-28869 as SPARK-28869 will create rolled
> event log files. This issue will be also on top of SPARK-29111 as SPARK-29111
> will add the ability to snapshot/restore from/to KVStore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
