[
https://issues.apache.org/jira/browse/KAFKA-19735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18028606#comment-18028606
]
kangning.li edited comment on KAFKA-19735 at 10/9/25 9:48 AM:
--------------------------------------------------------------
[~lianetm] Sorry for the late reply. I think in assign mode, apart from the
current consumer being able to modify the offset of the corresponding
TopicPartition, the Admin can also make modifications. Let’s assume such a
scenario: the offset cached by the consumer is 10, and all subsequent requests
to commit offset 10 will return successfully quickly without being sent to the
broker. At this point, if the Admin is used to set the offset to 11, the
consumer will not be aware of this change. As a result, the consumer caches an
invalid offset, which is inconsistent with expectations. WDYT ?
was (Author: JIRAUSER306086):
[~lianetm] Sorry for the late reply. I think in assign mode, apart from the
current consumer being able to modify the offset of the corresponding TP
(TopicPartition), the Admin can also make modifications. Let’s assume such a
scenario: the offset cached by the consumer is 10, and all subsequent requests
to commit offset 10 will return successfully quickly without being sent to the
broker. At this point, if the Admin is used to set the offset to 11, the
consumer will not be aware of this change. As a result, the consumer caches an
invalid offset, which is inconsistent with expectations. WDYT ?
> Add automatic commit offset caching in subscribe mode
> -----------------------------------------------------
>
> Key: KAFKA-19735
> URL: https://issues.apache.org/jira/browse/KAFKA-19735
> Project: Kafka
> Issue Type: Improvement
> Reporter: kangning.li
> Assignee: kangning.li
> Priority: Minor
>
> Suppose that no new messages have been written to the target topic for a long
> time, for example, the producer has not generated new data for nearly an
> hour. On the consumer side, when the user sets enable.auto.commit=true, the
> OFFSET_COMMIT request will still be repeatedly submitted every 5 seconds.
> This not only wastes network resources but also increases the burden on
> __consumer_offset.
>
> Therefore, maybe we can add a cache for committed offsets to check if the
> currently committed offset is consistent with the most recently successfully
> committed offset. If they are the same, this offset commit can be ignored. Of
> course, all the above applies to the subscribe mode.
>
> Advantages:
> * Reduce unnecessary network requests
> * Alleviate the processing pressure on the broker side
> * Relieve the pressure of log cleaning for __consumer_offset
>
> Disadvantage:
> * There may be hidden bugs that need to be discussed and identified
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
