[
https://issues.apache.org/jira/browse/SPARK-28781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CacheCheck updated SPARK-28781:
-------------------------------
Description: Once the update is called, newData is persisted at line 82.
However, only when the checkpoint is handling (satisfy the condition at line
94), the persist data is used for the second time (do checkpoint at line 97).
The other data which is not satisfied to the checkpoint condition is
unnecessary to be cached. The persistedQueue avoids too many unnecessary cached
data, but it is best to avoid every unnecessary persist operation. (was: *
Once the update is called, newData is persisted. However, only when the
checkpoint is handling, the persist data is used for the second time. The other
data which is not satisfied to the checkpoint condition is unnecessary to be
cached. The persistedQueue avoids too many unnecessary cached data, but it is
best to avoid every unnecessary persist operation.)
> Unneccesary persist in PeriodicCheckpointer.update()
> ----------------------------------------------------
>
> Key: SPARK-28781
> URL: https://issues.apache.org/jira/browse/SPARK-28781
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.3
> Reporter: CacheCheck
> Priority: Major
>
> Once the update is called, newData is persisted at line 82. However, only
> when the checkpoint is handling (satisfy the condition at line 94), the
> persist data is used for the second time (do checkpoint at line 97). The
> other data which is not satisfied to the checkpoint condition is unnecessary
> to be cached. The persistedQueue avoids too many unnecessary cached data, but
> it is best to avoid every unnecessary persist operation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
