Reason I'm asking this is that I don't want to maintain Brokers in DC1 as my consumers are exclusively in DC2.
On Mon, Feb 4, 2013 at 11:31 AM, Apoorva Gaurav <[email protected]>wrote: > Thanks Jun. > > So we'll have to maintain Zookeepers and Brokers in both the DCs while > Producers can be in DC1 and Consumers can be in target DC2. > > Are there any issues if we keep only Producer in DC1 talking > to Zookeepers and Brokers in DC2. I've been able to achieve this by making > a "hostname" entry in Broker properties which will have internal IP in DC2 > and public IP in DC1. > > > On Mon, Feb 4, 2013 at 10:55 AM, Jun Rao <[email protected]> wrote: > >> Apoorva, >> >> Kafka replication in 0.8 is designed for a Kafka cluster within the same >> DC. The following wiki describes cross DC mirroring using the tool >> MirrorMaker and how to optimize the throughput for long network latency. >> >> https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html >> >> Thanks, >> >> Jun >> >> On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <[email protected] >> >wrote: >> >> > Hello All, >> > >> > We are working on Kafka based event collection system. This needs to >> gather >> > events from across data centers. Lets say all the events will be >> produced >> > in DC1 while kafka brokers and consumers are lying in DC2. Round trip >> > between DC1 and DC2 can be around ~80 ms. Number of events should be >> around >> > ~50 million a day, peak being ~5K events a day, data volume ~100GB a >> day, >> > peak being ~10MB a day. What is the best way to do it. >> > >> > --- Is keeping the producer is DC1 sending events to DC2 a good idea. >> > --- Should my ZK quorum lie only in DC2 or should it spawn across both >> DC1 >> > and DC2. >> > --- Will this problem be solved easily in version .8.0 through broker >> > replication by keeping brokers in both DC1 and DC2. >> > >> > -- >> > Thanks & Regards, >> > Apoorvave >> > >> > > > > -- > Thanks & Regards, > Apoorva > -- Thanks & Regards, Apoorva
