[
https://issues.apache.org/jira/browse/KAFKA-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317516#comment-17317516
]
A. Sophie Blee-Goldman commented on KAFKA-12634:
------------------------------------------------
Oh. :/
By the way, shouldn't we also checkpoint _during_ a restore? We probably don't
want to do so at the same frequency as the commit interval, but it would be a
bummer if an app with an hour-long restore crashed in the middle and had to
start up again from scratch. Maybe we could pick an interval like 5min that
should allow users to avoid loosing all progress without severely slowing down
the restore even further?
> Should checkpoint after restore finished
> ----------------------------------------
>
> Key: KAFKA-12634
> URL: https://issues.apache.org/jira/browse/KAFKA-12634
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 2.5.0
> Reporter: Matthias J. Sax
> Priority: Major
>
> For state stores, Kafka Streams maintains local checkpoint files to track the
> offsets of the state store changelog topics. The checkpoint is updated on
> commit or when a task is closed cleanly.
> However, after a successful restore, the checkpoint is not written. Thus, if
> an instance crashes after restore but before committing, even if the state is
> on local disk the checkpoint file is missing (indicating that there is no
> state) and thus state would be restored from scratch.
> While for most cases, the time between restore end and next commit is small,
> there are cases when this time could be large, for example if there is no new
> input data to be processed (if there is no input data, the commit would be
> skipped).
> Thus, we should write the checkpoint file after a successful restore to close
> this gap (or course, only for at-least-once processing).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
