Well, if you have a cluster in each datacenter, all with the same topics, you can¹t just mirror the messages between them, as you will create a loop. The way we do it is to have a ³local² cluster and an ³aggregate² cluster. The local cluster has the data for only that datacenter. Then we run mirror makers that copy the messages from each of the local clusters into the aggregate cluster. Everything produces into the local clusters, and nothing produces into the aggregate clusters. In general, consumers consume from the aggregate cluster (unless they specifically want only local data).
The mirror maker is as fault tolerant as any other consumer. That is, if a mirror maker goes down, the others configured with the same consumer group (we generally run at least 4 for any mirror maker, sometimes up to 10) will rebalance and start back up from the last committed offset. What you need to watch out for is if the mirror maker is unable to produce messages, for example, if the network goes down. If it can still consume messages, but cannot produce them, you will lose messages as the consumer will continue to commit offsets with no knowledge that the producer is failing. -Todd On 5/8/14, 11:20 AM, "Weide Zhang" <[email protected]> wrote: >Hi, > >I have a question about mirror maker. say I have 3 data centers each >producing topic 'A' with separate kafka cluster running. if 3 of the data >need to be kept in sync with each other, shall i create 3 mirror maker in >each data center to get the data from the other two ? > >also, it mentioned that mirror making is not fault tolerant ? so what will >be the behavior of mirror consumer if it went down due to network and back >up ? do they catch up with last offset from which they last mirror ? If >so, >is it enabled by default or I have to configure ? > >Thanks a lot, > >Weide
