[
https://issues.apache.org/jira/browse/SAMZA-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hai Lu updated SAMZA-2356:
--------------------------
Fix Version/s: 1.3
> Do not trim changelog if time since last checkpoint is greater than
> min.compaction.lag
> --------------------------------------------------------------------------------------
>
> Key: SAMZA-2356
> URL: https://issues.apache.org/jira/browse/SAMZA-2356
> Project: Samza
> Issue Type: Task
> Reporter: Prateek Maheshwari
> Assignee: Prateek Maheshwari
> Priority: Major
> Fix For: 1.3
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In the current implementation for transactional state, if the checkpointed
> changelog offset gets compacted, (e.g. if the processor was down for more
> than min compaction lag), and the checkpointed changelog offset is b/w the
> oldest and newest post compaction, we will end up deleting part of the
> current state (potentially everything after checkpointed offset). We should
> not do this.
> Instead, we should write the checkpoint timestamp along with the the
> checkpointed changelog offset, and not do a trim during restore if time since
> last checkpoint is older than (or close to) min.compaction.lag. We should
> instead restore to latest and log a warning that transactional state
> guarantee doesn't hold (i.e., same as the current behavior). The fix for
> users would then be to increase the min.compaction.lag.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
