Apologies for bumping my own post, but really hoping someone has experience with MirrorMaker who might be able to help with my question below. :)
Thanks! Chris On Fri, Oct 27, 2017 at 10:59 AM, Chris Neal <cwn...@gmail.com> wrote: > Hi all, > > I've been struggling with this for awhile and need some help please. :) > > I have a source cluster running Kafka 0.8.1.1 with about 900 topics. > I have a target cluster running Kafka 0.10.2 with those same topics > pre-created with the same number of partitions per topic as the source > cluster. > > I'm running Mirror Maker from the 0.8.1.1 Kafka version, and cannot seem > to get the produced records to go anywhere besides partition 0 of all these > topics. :( > > I found this page to be helpful WRT performance tuning, but it did not > solve my partition producer problem: > > https://community.hortonworks.com/articles/79891/kafka- > mirror-maker-best-practices.html > > My consumer.config: > ====================================== > zookeeper.connect=zk1:2181,zk2:2181,zk3:2181 > group.id=prod_perf_mirrormaker2_new1 > num.consumer.fetchers=1100 > client.id=MirrorMakerConsumer_perf > ====================================== > > My producer config: > ====================================== > metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092 > compression.codec=gzip > producer.type=async > batch.num.messages=1000 > message.send.max.retries=2 > client.id=MirrorMakerProducer_perf > ====================================== > > My start script: > ====================================== > #!/bin/sh > > export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G" > > /home/chris.neal/kafka_2.9.2-0.8.1.1/bin/kafka-run-class.sh > kafka.tools.MirrorMaker \ > --producer.config /home/chris.neal/mirror_maker/prod_perf/producer.properties > \ > --consumer.config /home/chris.neal/mirror_maker/prod_perf/consumer.properties > \ > --num.streams=10 \ > --whitelist '<snip long list of about 60 topics>' > ====================================== > > I do know that there is some combination of "num.streams" and > "num.consumer.fetchers" that should get it working, but I can't figure out > what that combination is. > > All topics have the same setup of 10 partitions per topic. > > Can anyone shed some light on how to make this go across all 10 > partitions, and not just partition 0? > > Thank you SO much for your time and help. > Chris >
