[
https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472427#comment-15472427
]
Patrick Te Tau commented on KAFKA-1006:
---------------------------------------
Hi [~guozhang], we are also having trouble with our integration tests.
I can manually set the offset for new topics but this will break my
subscription to pre-existing topics. Because have no way of telling whether the
topic is a new one or an old one, I have no way to switch my strategy.
I have considered storing a list of topics on my client but this solution fails
when I run multiple clients.
Any suggestions?
> Consumer loses messages of a new topic with auto.offset.reset = largest
> -----------------------------------------------------------------------
>
> Key: KAFKA-1006
> URL: https://issues.apache.org/jira/browse/KAFKA-1006
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Swapnil Ghike
> Assignee: Guozhang Wang
> Labels: usability
>
> Consumer currently uses auto.offset.reset = largest by default. If a new
> topic is created, consumer's topic watcher is fired. The consumer will first
> finish partition reassignment as part of rebalance and then start consuming
> from the tail of each partition. Until the partition reassignment is over,
> the server may have appended new messages to the new topic, consumer won't
> consume these messages. Thus, multiple batches of messages may be lost when a
> topic is newly created.
> The fix is to start consuming from the earliest offset for newly created
> topics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
