Not sure exactly what's happening there. In any case, 0.7 is really old.
You probably want to upgrade to the latest 0.8 release.
Thanks,
Jun
On Tue, Jan 6, 2015 at 12:16 AM, tao li tust.05102...@gmail.com wrote:
Hi,
We are using mirrormaker to replicate data between two kafka clusters in
difference datacenter, a fiber connected them, 1G bandwidth, ~35ms
lantency.
we found that the MessageIn in JMX beans with the broker in destination
cluster greater than broker in source cluster.
Our configuration like this:
destination source cluster both deployed kafka 0.7.0
source clusters has 6 brokers with 32 partitions each,target clusters has
12 brokers with 32 partitions each and 6 mirrormakers doing
replication,replicate only one topic.
mirrormaker configuration like this:
--num.streams 128
--num.producers 16
=
consumer.properties:
shallowiterator.enable=true
socket.buffersize=31457280
fetch.size=1048576
=
producer.properties:
queue.enqueueTimeout.ms http://queue.enqueuetimeout.ms/=-1
# the number of messages batched at the producer
batch.size=4000
#default 5000
connect.timeout.ms=1
#default 3
socket.timeout.ms=30
==
test result:
MessageIn: source cluster = 252,560,169, target cluster = 253,128,669
another problem, sometimes one or more broker in source clusters has not be
consumered during the mirrormakers replicate other brokers well.
Does anyone have an idea to resolve them.
