[
https://issues.apache.org/jira/browse/SPARK-28867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950307#comment-16950307
]
wuyi commented on SPARK-28867:
------------------------------
[~irashid] Thanks for putting this as a related issue to SPARK-20656.
Yeah, I agree that SPARK-28867 and SPARK-20656 is not quite the same. But
they both want to implement incremental replay basing on snapshot
mechanism(SPARK-20656 said it will have UI data stored on disk). Actually, I
think SPARK-20656 is more similar to SPARK-28594. As both SPARK-20656 and
SPARK-28594 plan to do snapshot on SHS, while in SPARK-28867 it plans to do
snapshot in AppStatusListener(driver side).
> InMemoryStore checkpoint to speed up replay log file in HistoryServer
> ---------------------------------------------------------------------
>
> Key: SPARK-28867
> URL: https://issues.apache.org/jira/browse/SPARK-28867
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: wuyi
> Priority: Major
>
> HistoryServer now could be very slow to replay a large log file at the first
> time and it always re-replay an inprogress log file after it changes. we
> could periodically checkpoint InMemoryStore to speed up replay log file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
