I don't know if I'd go as far as "elegant". Functional, definitely. :)
Yang, I'm not entirely sure what you're looking for here. You can already specify the acks setting in the producer if you do not care about acknowledgement from your remote produce requests (setting it to 0). If you do care, you just have to tolerate the latency. The speed of light isn't just a good idea, it's the law. -Todd On Tue, Mar 3, 2015 at 3:34 PM, Jeff Schroeder <jeffschroe...@computer.org> wrote: > Mirror maker is about separating latency and failure domains. I think it is > a very elegant solution to a difficult problem. My suspicion is that the > LinkedIn / Confluent team agrees. > > On Tue, Mar 3, 2015 at 3:50 PM, Yang <teddyyyy...@gmail.com> wrote: > > > thanks guys. > > > > it's just quite a lot of ops cost to setup and monitor a separate > cluster, > > connected through mirror maker. sometimes if I have just a single > > producer/consumer in a new cluster, it would be more desirable to just > > connect it directly to an existing kafka setup. > > > > I remember at least in an earlier version of Cassandra (about 2 years > ago, > > when I still actively worked on that), you could define network > topologies > > , and define the different acknowledgement requirements for different > parts > > of the topology. if this kind of flexibility is built into the internal > > protocol of Kafka, it would be much nicer. personally I think the mirror > > maker is more of a patch work than a systematic design. > > > > On Mon, Mar 2, 2015 at 7:53 AM, Todd Palino <tpal...@gmail.com> wrote: > > > > > Latencies like this are one of the big reasons that we run our Kafka > > > clusters local to the producers and consumers. Another is network > > > partition. As Jeff noted, mirror maker is the way to connect them > > together. > > > Our architecture uses a local cluster in each datacenter, and then an > > > aggregate cluster (either in those datacenters or in another) where we > > > mirror all the traffic from the local clusters together into one place. > > > This way clients can pick a local or a global view of the data, and we > > can > > > have everyone still stay in their local Kafka clusters. > > > > > > -Todd > > > > > > > > > On Fri, Feb 27, 2015 at 4:21 PM, Yang <teddyyyy...@gmail.com> wrote: > > > > > > > we tested our new application that reads and writes to kafka. > > > > > > > > at first we found the access latency is very high. then we realized > > that > > > > it's because the client and server are in different colos. moving > them > > > > together reduces down the access time to < 4 ms. > > > > > > > > > > > > I was wondering if there are any techniques/properties to set , or > > maybe > > > > enhancements to be made to the kafka code, that could cater to such > > > > situations ? > > > > > > > > Thanks > > > > Yang > > > > > > > > > > > > > -- > Jeff Schroeder > > Don't drink and derive, alcohol and analysis don't mix. > http://www.digitalprognosis.com >
