[
https://issues.apache.org/jira/browse/KAFKA-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111675#comment-15111675
]
Joel Koshy commented on KAFKA-2758:
-----------------------------------
Unless I'm misunderstanding I don't think we should do (1) - i.e., ideally the
"age" of the commit should be same for all partitions. So if you fail to commit
offsets for some partitions those offsets could get compacted out early (and
possibly removed entirely). See KAFKA-1510
> Improve Offset Commit Behavior
> ------------------------------
>
> Key: KAFKA-2758
> URL: https://issues.apache.org/jira/browse/KAFKA-2758
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Guozhang Wang
> Labels: newbiee
> Fix For: 0.9.0.1
>
>
> There are two scenarios of offset committing that we can improve:
> 1) we can filter the partitions whose committed offset is equal to the
> consumed offset, meaning there is no new consumed messages from this
> partition and hence we do not need to include this partition in the commit
> request.
> 2) we can make a commit request right after resetting to a fetch / consume
> position either according to the reset policy (e.g. on consumer starting up,
> or handling of out of range offset, etc), or through the {code} seek {code}
> so that if the consumer fails right after these event, upon recovery it can
> restarts from the reset position instead of resetting again: this can lead
> to, for example, data loss if we use "largest" as reset policy while there
> are new messages coming to the fetching partitions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
