[
https://issues.apache.org/jira/browse/KAFKA-18168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17921970#comment-17921970
]
Matthias J. Sax commented on KAFKA-18168:
-----------------------------------------
Hey, thanks for getting back. All good. No need to apologize for anything.
Q1: Yes, that sounds right to me.
Q2: Would need to look into more details to answer precisely. The method
itself, does make a decision if we need to checkpoint or not (and yes, you are
right, it check for the 10K threshold); but it just returns a boolean about the
decision but does not do the checkpoint itself. Furthermore, the caller can
pass in `enforceCheckpoint` flag. Hence, overall the changes could also be a
mix of changes inside this method (eg, add some configurable checkpoint time
interval in addition to the 10K threshold – or maybe even remove the hardcode
10K threshold – or allow for config for the existing threshold), plus changing
the code which calls this method (ie, `StateConsumer.close()` or
`GlobalStateUpdateTask.close()`, we would know we alway want to checkpoint,
and we pass `enforceCheckpoint=true` when using the method). Does this
(somewhat fuzzy) answer help?
> GlobalKTable does not checkpoint restored offsets until next 10K events
> -----------------------------------------------------------------------
>
> Key: KAFKA-18168
> URL: https://issues.apache.org/jira/browse/KAFKA-18168
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 3.4.1, 3.8.1
> Reporter: Sergey Zyrianov
> Assignee: Janindu Pathirana
> Priority: Minor
>
> As in https://issues.apache.org/jira/browse/KAFKA-5241, there is a state of
> considerable size kept on a topic that backs up GlobalKTalbe. Restoring
> GlobalKTable takes minutes before it is operational. After successful restore
> the checkpoint file is not created until further 10K events happen on the
> topic.
> The following scenario illustrates the issue:
> # {*}Scaling Out{*}: When a new instance (e.g., pod X) is added to an
> already running set of instances (pods 0...X-1), the new instance will
> restore the state successfully. However, it will not create a checkpoint file
> until 10K events are processed on the {{GlobalKTable}} topic.
> # {*}Lack of Traffic{*}: If there is no new traffic on the {{GlobalKTable}}
> topic, there is no mechanism to force the creation of the checkpoint file.
> The state remains uncheckpointed. Ref
> [https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StateManagerUtil.java#L78C35-L78C72]
> # {*}Instance Restart{*}: If the new instance (pod X) is restarted (due to
> update for ex) before 10K events have been processed, it will have to restore
> the entire state from the topic again, leading to the same time-consuming
> restoration process. This issue persists across restarts.
> IMO, checkpointing during the restore process and upon completion/close is
> missing in the current implementation
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
