Hi Umesh, I am new to kafka as well, and configuring the MirrorMaker. I got mine working in the following way.
I run the mirror maker instance on the mirror cluster, as in where you want to mirror the topics to, although I'm not sure it matters. I use the following options when starting my service (systemd file): KAFKA_RUN="/opt/kafka/bin/kafka-run-class.sh" KAFKA_ARGS="kafka.tools.MirrorMaker" KAFKA_CONFIG="--new.consumer --offset.commit.interval.ms=5000 --consumer.config /opt/kafka/config/consumer-mirror1.properties --producer.config /opt/kafka/config/producer.properties --whitelist=\".*\"" Without the --new.consumer parameter, the --consumer.config and producer.config files need to contain the zookeeper config for relevant clusters. When using the --new.consumer switch this is no longer needed (as I understand it). The consumer config points at my source cluster, the producer config points locally, to my mirror cluster. I think it's also important to configure the whitelist to tell it which topics you want to mirror, in my case I mirror all of them with a wildcard. Not much config in the consumer.config and producer.config files apart from the bootstrap.servers list, pointing at the relevant cluster. I have 3 brokers in my mirror cluster and each one of them runs the same mirror maker service so one will take over if another one fails. I hope someone will correct me if I am wrong about anything, and hopefully this will help! Cheers! On Thu, Aug 25, 2016 at 9:36 AM, UMESH CHAUDHARY <[email protected]> wrote: > Hey Folks, > I was trying to understand the behavior of MirrorMaker but looks like I am > missing something here. Please see the steps which I performed : > > 1) I configured MM on source Kafka cluster > 2) Created a topic and pushed some data in it using console producer. > 3) My understanding is that MM would start mirroring the data (which is > there in the topic) based on "offsetCommitIntervalMs" and it would be there > in destination cluster. > > https://github.com/apache/kafka/blob/0.9.0/core/src/ > main/scala/kafka/tools/MirrorMaker.scala#L503 > > 4) But when I list the topics on destination, I cant see the topic which I > recently created on source. > 5) I tried to check the offset of "mirrormaker_group" for that topic (on > source cluster) using kafka.admin.ConsumerGroupCommand, I see the offsets > for that topic as "unknown". > 6) But when I start console consumer for that topic on source or > destination (auto creation of topic is true), I see that all data in being > mirrored via MM and kafka.admin.ConsumerGroupCommand tells the right > offsets this time. > > Is this expected behavior of MM or did I mess up with some configuration? > > Regards, > Umesh >
