[
https://issues.apache.org/jira/browse/KAFKA-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211870#comment-14211870
]
Jiangjie Qin commented on KAFKA-345:
------------------------------------
[~nehanarkhede] It is mainly because recently we have many mirror maker issue
that leads to a hard kill, which cause data loss. So we want to eliminate the
data loss in mirror maker as soon as possible. KAFKA-1650 is opened for this
issue. In order to do that, we need to turn off auto offset commit. That means
we are almost guaranteed to have duplicates on consumer rebalance. So we want
to add a callback to avoid duplicates on consumer rebalance.
> Add a listener to ZookeeperConsumerConnector to get notified on rebalance
> events
> --------------------------------------------------------------------------------
>
> Key: KAFKA-345
> URL: https://issues.apache.org/jira/browse/KAFKA-345
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.7, 0.8.0
> Reporter: Peter Romianowski
> Attachments: KAFKA-345.patch, KAFKA-345.patch
>
>
> A sample use-case
> In our scenario we partition events by userid and then apply these to some
> kind of state machine, that modifies the actual state of a user. So events
> trigger state transitions. In order to avoid the need of loading user's state
> upon each event processed, we cache that. But if a user's partition is moved
> to another consumer and then back to the previous consumer we have stale
> caches and hell breaks loose. I guess the same kind of problem occurs in
> other scenarios like counting numbers by user, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
