Thanks Jun. We have a cluster as well. However, in our setup each Kafka node has both the mirror maker and broker process running, with the mirror maker writing only to the local broker. The main rationale was to limit further use of network. But, I guess that wasn't a good idea given the failure scenarios.
On Fri, Mar 22, 2013 at 8:58 PM, Jun Rao <jun...@gmail.com> wrote: > Yes, this is true if you have only 1 broker in the target cluster. If you > set up multiple brokers in the target cluster, mirror maker will send > messages to available brokers. > > Thanks, > > Jun > > On Fri, Mar 22, 2013 at 12:02 PM, Riju Kallivalappil < > riju.kallivalap...@corp.247customer.com> wrote: > > > Hi, > > I've a question about fault tolerance of Kafka mirror maker (0.7.1) > > > > Let's say that I've a mirroring setup with topics in broker B1 mirrored > to > > broker B2. > > > > On B2, I've Kafka mirror maker and Kafka broker process running. Now, > > following is what I noticed when the Kafka broker process on B2 is > > restarted. > > > > The embedded consumer in the Mirror Maker continues to consume messages > > from B1 and updates the ZK offsets even after the Kafka process is > stopped. > > Pretty much all of the messages that are consumed during the time that > the > > local broker process is down is lost. > > > > Is this expected behavior? Looking through the code, I suppose it is. > > > > Are there any suggestions on how to avoid this kind of data loss? > > > > Thanks, > > Riju > > >
