It is possible to implement offset-preserving mirrors - however, that would work only if there is one source cluster mirroring into target cluster(s) (as opposed to mirroring multiple source clusters into one target cluster). Anyway, as Jun said right now you have to either read from the tail or head of the fail-over cluster.
Thanks, Joel On Mon, Sep 23, 2013 at 8:53 PM, Jun Rao <jun...@gmail.com> wrote: > Tom, > > MirrorMaker is nothing but a tool with an embedded consumer and producer. > So, mirroring is one-way. The offsets btw the source and the target > clusters are not preserved though. If the consumer is real time, during > failover, you can start the consumer in the failover site by consuming from > the end of the Kafka topic. The impact is that a small number of messages > during the failover window could be lost and the consumer may get a small > number of duplicates too. > > Thanks, > > Jun > > > On Mon, Sep 23, 2013 at 4:53 PM, Tom Amon <ta46...@gmail.com> wrote: > >> Hi All, >> >> I am hoping that mirror maker is what I'm looking for. I would like to have >> complete data center fail over from one kafka cluster to another. If one >> data center goes down, my producers and consumers will start using the >> mirrored cluster until the primary is back online. Is that something I can >> do with mirror maker? >> >> I'm thinking of using a short TTL on the DNS entries for ZooKeeper and the >> brokers and have a monitor switch them to the mirrored cluster if we detect >> that the primary data center has gone offline. >> >> Some specific questions: >> >> 1) Does mirror maker replicate the consumer offsets from the source >> cluster? In other words, can a consumer safely fail over from the source >> cluster to the mirrored cluster? >> >> 2) I assume that when the primary comes back online, all messages published >> to the mirror cluster are not back published to the primary cluster, >> correct? The mirroring is one way only? >> >> 3) Same goes for consumer offsets (if they are mirrored) in the mirrored >> zookeeper. A consumer will have issues going back to the primary cluster >> since the offsets will be different, correct? Even if the offsets are >> mirrored I'm not sure how this would work to switch back to the primary >> cluster. Maybe when the primary comes back online switch the mirror maker >> to make the current cluster the source and the now back online cluster the >> mirror? >> >> Any other way that any one has used to do complete data center fail over? >> I've read that doing ZooKeeper cross data center is not advised, so minus >> some trickery that's out. Similarly for doing a single cluster of brokers >> cross data center (in response to a previous question I posted here). >> >> Thanks. >>
