Mirror maker is about separating latency and failure domains. I think it is a very elegant solution to a difficult problem. My suspicion is that the LinkedIn / Confluent team agrees.
On Tue, Mar 3, 2015 at 3:50 PM, Yang <[email protected]> wrote: > thanks guys. > > it's just quite a lot of ops cost to setup and monitor a separate cluster, > connected through mirror maker. sometimes if I have just a single > producer/consumer in a new cluster, it would be more desirable to just > connect it directly to an existing kafka setup. > > I remember at least in an earlier version of Cassandra (about 2 years ago, > when I still actively worked on that), you could define network topologies > , and define the different acknowledgement requirements for different parts > of the topology. if this kind of flexibility is built into the internal > protocol of Kafka, it would be much nicer. personally I think the mirror > maker is more of a patch work than a systematic design. > > On Mon, Mar 2, 2015 at 7:53 AM, Todd Palino <[email protected]> wrote: > > > Latencies like this are one of the big reasons that we run our Kafka > > clusters local to the producers and consumers. Another is network > > partition. As Jeff noted, mirror maker is the way to connect them > together. > > Our architecture uses a local cluster in each datacenter, and then an > > aggregate cluster (either in those datacenters or in another) where we > > mirror all the traffic from the local clusters together into one place. > > This way clients can pick a local or a global view of the data, and we > can > > have everyone still stay in their local Kafka clusters. > > > > -Todd > > > > > > On Fri, Feb 27, 2015 at 4:21 PM, Yang <[email protected]> wrote: > > > > > we tested our new application that reads and writes to kafka. > > > > > > at first we found the access latency is very high. then we realized > that > > > it's because the client and server are in different colos. moving them > > > together reduces down the access time to < 4 ms. > > > > > > > > > I was wondering if there are any techniques/properties to set , or > maybe > > > enhancements to be made to the kafka code, that could cater to such > > > situations ? > > > > > > Thanks > > > Yang > > > > > > -- Jeff Schroeder Don't drink and derive, alcohol and analysis don't mix. http://www.digitalprognosis.com
