Jungtaek Lim created SPARK-29579:
------------------------------------
Summary: Guarantee compatibility of snapshot (live entities,
KVstore entities)
Key: SPARK-29579
URL: https://issues.apache.org/jira/browse/SPARK-29579
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Jungtaek Lim
This issue is a follow-up issue after SPARK-29111 and SPARK-29261, which both
issues WILL NOT guarantee compatibility.
To safely clean up old event log files after snapshot has been written for
these files, we have to ensure the snapshot file can restore the state as same
as we replay from these event log files. The issue is on migrating to the newer
Spark version - if snapshot is not readable due to incompatibility, the app
cannot be read entirely as we've already removed old event log files. If we can
guarantee compatibility we can move on to the next item, cleaning up old event
log files to save space.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
