[
https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapnil Ghike updated KAFKA-1006:
---------------------------------
Description:
Consumer currently uses auto.offset.reset = largest by default. If a new topic
is created, consumer's topic watcher is fired. The consumer will first finish
partition reassignment as part of rebalance and then start consuming from the
tail of each partition. Until the partition reassignment is over, the server
may have appended new messages to the new topic, consumer won't consume these
messages. Thus, multiple batches of messages may be lost when a topic is newly
created.
The fix is to start consuming from the earliest offset for newly created topics.
was:
Mirror maker currently uses auto.offset.reset = largest on the consumer side by
default. If a new topic is created, consumer's topic watcher is fired. The
consumer will first finish partition reassignment as part of rebalance and then
start consuming from the tail of each partition. Until the partition
reassignment is over, the server may have appended new messages to the new
topic, mirror maker won't consume these messages. Thus, multiple batches of
messages may be lost when a topic is newly created.
The fix is to start consuming from the earliest offset for newly created topics.
> Mirror maker loses messages of a new topic
> ------------------------------------------
>
> Key: KAFKA-1006
> URL: https://issues.apache.org/jira/browse/KAFKA-1006
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Swapnil Ghike
>
> Consumer currently uses auto.offset.reset = largest by default. If a new
> topic is created, consumer's topic watcher is fired. The consumer will first
> finish partition reassignment as part of rebalance and then start consuming
> from the tail of each partition. Until the partition reassignment is over,
> the server may have appended new messages to the new topic, consumer won't
> consume these messages. Thus, multiple batches of messages may be lost when a
> topic is newly created.
> The fix is to start consuming from the earliest offset for newly created
> topics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
