Hi All, I am hoping that mirror maker is what I'm looking for. I would like to have complete data center fail over from one kafka cluster to another. If one data center goes down, my producers and consumers will start using the mirrored cluster until the primary is back online. Is that something I can do with mirror maker?
I'm thinking of using a short TTL on the DNS entries for ZooKeeper and the brokers and have a monitor switch them to the mirrored cluster if we detect that the primary data center has gone offline. Some specific questions: 1) Does mirror maker replicate the consumer offsets from the source cluster? In other words, can a consumer safely fail over from the source cluster to the mirrored cluster? 2) I assume that when the primary comes back online, all messages published to the mirror cluster are not back published to the primary cluster, correct? The mirroring is one way only? 3) Same goes for consumer offsets (if they are mirrored) in the mirrored zookeeper. A consumer will have issues going back to the primary cluster since the offsets will be different, correct? Even if the offsets are mirrored I'm not sure how this would work to switch back to the primary cluster. Maybe when the primary comes back online switch the mirror maker to make the current cluster the source and the now back online cluster the mirror? Any other way that any one has used to do complete data center fail over? I've read that doing ZooKeeper cross data center is not advised, so minus some trickery that's out. Similarly for doing a single cluster of brokers cross data center (in response to a previous question I posted here). Thanks.
