[
https://issues.apache.org/jira/browse/KAFKA-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332534#comment-15332534
]
James Clarke commented on KAFKA-3848:
-------------------------------------
Thanks for the quick rely. I think you might be right with
{{auto.offset.reset}}. I'll perform some testing tonight.
I consumed from dc1 after running the full test and with {{--from-beginning}}.
This is not isolated to wildcard topics because I produced directly to 10
topics which were explicitly listed in the whitelist. I think it is just an
edge case with newly created topics (via {{auto.create.topics.enable}}).
Assuming the issue with {{auto.offset.reset}} default behavior, and the
consumer is detecting the new topic after the first message is produced (and
therefore consuming from offset 1 onwards) then I suggest the documentation be
updated to point out that {{auto.offset.reset}} should be set to {{smallest}}.
Currently:
{quote}
Combining mirroring with the configuration auto.create.topics.enable=true makes
it possible to have a replica cluster that will automatically create and
replicate all data in a source cluster even as new topics are added.
{quote}
> Mirror Maker sometimes fails to mirror first batch of messages
> --------------------------------------------------------------
>
> Key: KAFKA-3848
> URL: https://issues.apache.org/jira/browse/KAFKA-3848
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.1, 0.10.0.0
> Reporter: James Clarke
>
> I am seeing an intermittent issue in Mirror Maker where the first batch of
> messages are not always mirrored to the target cluster. All messages after
> the first batch are mirrored.
> I have a github repo
> ([jc/kafka-mirror-maker-test|https://github.com/jc/kafka-mirror-maker-test])
> which reproduces the issue using Confluent's docker containers (running
> 0.10.0). However on our environment we are using our own kafka containers
> running 0.9.0.1.
> Environment:
> - edge datacenter dc1. 1 zk server, 1 kafka server.
> - aggregate datacenter dc2. 1 zk server, 1 kafka server, 1 mirror maker.
> - kafka server setup to auto create topics
> - Mirrror maker configured mirror from dc1 to dc2 using a whitelist
> containing both explicitly topics and regex topics.
> Steps to reproduce:
> - Send message to a non-existent topic in dc1.
> - Send a second message to topic in dc1.
> Observed:
> - After first message the topic is not created in dc2.
> - After second message topic is present in dc2.
> - Consuming from topic in dc1 shows both messages.
> - Consuming from topic in dc2 shows only the second message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
