Then it's much simpler, just use replication-factor=3 for your topics,
either by setting it when creating topics manually, or by
setting default.replication.factor=3 in the broker settings. Kafka will
then take care of the leadership on the partitions so both the load is
balanced, and in case some broker fails, the clients will connect to other
brokers.
The mirror maker tool is only for copying data between clusters.
On Tue, Jun 28, 2016 at 2:50 PM cs user <acldstk...@gmail.com> wrote:
> Hi there,
>
> I mean 1 cluster with 3 nodes. So I will need to run the mirror maker
> cluster on each of the 3 nodes in the cluster, in case of the loss of a
> node, the other 2 will continue to pull messages off the consumer cluster.
> It does seem to work correctly when I tested it. It just warns about topics
> with only one partition, when multiple clients are trying to consume from
> it:
>
> "No broker partitions consumed by consumer thread"
>
> One problem I have found is that when a topic is created, at first the
> mirror is unable to consume messages instantly. It seems that only say 70%
> of the messages( say 7,000 of of 10,000) that were sent to newly created
> topic make it to the mirror.
>
> After the first batch, however, the messages seem to be mirrored correctly.
>
> I checked the mirror maker process logs and found the following, just as
> the topic was created:
>
> [2016-06-28 11:56:46,339] WARN Error while fetching metadata with
> correlation id 0 : {topictest4=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-28 11:56:46,545] WARN Error while fetching metadata with
> correlation id 1 : {topictest4=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-28 11:56:46,649] WARN Error while fetching metadata with
> correlation id 2 : {topictest4=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient)
>
> Is there any reason not all of the messages made it through? Is there a way
> to reset the offset so it tries to sync the messages from the beginning?
>
> Thanks!
>
> On Tue, Jun 28, 2016 at 12:00 PM, Gerard Klijs <gerard.kl...@dizzit.com>
> wrote:
>
> > With 3 nodes, I assume you mean 3 clusters? If I understand correctly,
> say
> > you have 3 clusters, A, B, and C, you simultaneously:
> > - want to copy from A and B to C, to get an aggregation in C
> > - want to copy fram A and C to B, to get a fail-back aggregation in B.
> > Now what will happen when a message is produced in cluster a?
> > - it will be copied to both C and B.
> > - the copy wil cause a new copy in C and B,
> > etc.
> > There are several ways out if this, depending on your use case. It's
> pretty
> > easy to change the behaviour of the mirrormaker, for example to copy it
> to
> > $topic-aggregation instead of $topic, and to not copy it when the topic
> > ends with aggregation
> >
> > On Tue, Jun 28, 2016 at 10:15 AM cs user <acldstk...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > So I understand I can run the following to aggregate topics from two
> > > different clusters into a mirror:
> > >
> > > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > > sourceCluster1Consumer.config --consumer.config
> > > sourceCluster2Consumer.config --num.streams 2 --producer.config
> > > targetClusterProducer.config --whitelist=".*"
> > >
> > > Lets say my kafka mirror cluster consists of 3 nodes, can the above
> > process
> > > be started on each of the 3 nodes, so that in the event it fails on one
> > the
> > > other 2 will keep going?
> > >
> > > Or should only one of the nodes attempt to perform the aggregation?
> > >
> > > Thanks!
> > >
> >
>
