I've attached my two configs here. Pay close attention to the --num-streams argument to mirror-maker. I have a lot of thru-put on my topics so I ended up matching the number of streams = number of partitions for each of my topics. A stream is essentially just a consumer and producer thread. If you have less streams/threads than number of partitions * topics replicated, it seems like mirror-maker roundrobins the threads across the partitions/topics, which may be fine based on your ingest rate and network latency between clusters.
Additionally, I had to raise the java heap size considerably to prevent mirror-maker from OOMing. You can check the status of mirror maker and get the lag by running: kafka-consumer-offset-checker.sh --topic "your-topic-here" --zookeeper src-zk-host:2181 --group mirrormaker-consumer-id-here I *think* I also edited config/tools-log4j.properties and made mirror-maker log info level messages by changing the first line to read as follows. This allowed me to more easily debug issues and fine tune my settings. log4j.rootLogger=INFO, fileAppender, stderr consumer.properties ################## # Zookeeper connection string # comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002" zookeeper.connect="source-zk-host1:2181,source-zk-host2:2181" # timeout in ms for connecting to zookeeper zookeeper.connection.timeout.ms=6000 #consumer group id group.id=mirrormaker-consumer-id-here producer.properties ################## # list of brokers used for bootstrapping knowledge about the rest of the cluster # format: host1:port1,host2:port2 ... bootstrap.servers=dest-kafka-host1:9092,dest-kafka-host2:9092 # name of the partitioner class for partitioning events; default partition spreads data randomly (in most cases) ## I needed to use a custom partitioner to get my data to be distributed evenly, but you'll probably ## be fine leaving this commented out. see jira kafka-3333 if you're curious to what I used. # partitioner.class= # specify the compression codec for all data generated: none, gzip, snappy, lz4. # the old config values work as well: 0, 1, 2, 3 for none, gzip, snappy, lz4, respectively compression.codec=none ## Increment size of batches batch.size:65536 client.id: mirror-maker On Wed, Mar 9, 2016 at 5:43 AM, prabhu v <prabhuvrajp...@gmail.com> wrote: > Thanks for the reply.. > > I will remove the bootstrap.servers property and add zookeeper.connect in > consumer properties and let you know.... > > Also, is there any way we can check how much data the target data center is > lagging behind source DC? > > > On Wed, Mar 9, 2016 at 3:41 PM, Gerard Klijs <gerard.kl...@dizzit.com> > wrote: > > > What do you see in the logs? > > It could be it goes wrong because you have the bootstrap.servers property > > which is not supported for the old consumer. > > > > On Wed, Mar 9, 2016 at 11:05 AM Gerard Klijs <gerard.kl...@dizzit.com> > > wrote: > > > > > Don't know the actual question, it matters what you want to do. > > > Just watch out trying to copy every topic using a new consumer, cause > > then > > > internal topics are copied, leading to errors. > > > Here is a temple start script we used: > > > > > > #!/usr/bin/env bash > > > export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true > > -Dcom.sun.management.jmxremote.authenticate=false > > -Dcom.sun.management.jmxremote.ssl=false > > -Dcom.sun.management.jmxremote.local.only=false > > -Djava.rmi.server.hostname=<NODE_HOSTNAME> > > -Dcom.sun.management.jmxremote.rmi.port=<MM_JMX_PORT>" > > > export JMX_PORT=<MM_JMX_PORT> > > > /usr/bin/kafka-mirror-maker --consumer.config > > $HOME_DIR/consumer.properties --producer.config > > $HOME_DIR/producer.properties --whitelist='<REGEX_MM>' 1>> > > $LOG_DIR/mirror-maker.log 2>> $LOG_DIR/mirror-maker.log > > > > > > Both the consumer and producer configs have sensible defaults, these > are > > out consumer.properties template: > > > > > > #Consumer template to be used with the mirror maker > > > zookeeper.connect=<CONSUMER_ZOOKEEPERS> > > > group.id=mirrormaker > > > auto.offset.reset=smallest > > > #next property is not available in new consumer > > > exclude.internal.topics=true > > > > > > *And a producer.properties template:* > > > > > > #Producer template to be used with the mirror maker > > > bootstrap.servers=<PRODUCER_BROKERS> > > > client.id=mirrormaker > > > > > > Because the internal topics can't be excluded in the new consumer yet, > > we use the old consumer. > > > > > > Hope this helps. > > > > > > > > > On Wed, Mar 9, 2016 at 10:57 AM prabhu v <prabhuvrajp...@gmail.com> > > wrote: > > > > > >> Hi Experts, > > >> > > >> I am trying to mirror > > >> > > >> > > >> > > >> > > >> -- > > >> Regards, > > >> > > >> Prabhu.V > > >> > > > > > > > > > -- > Regards, > > Prabhu.V >
