Latencies like this are one of the big reasons that we run our Kafka clusters local to the producers and consumers. Another is network partition. As Jeff noted, mirror maker is the way to connect them together. Our architecture uses a local cluster in each datacenter, and then an aggregate cluster (either in those datacenters or in another) where we mirror all the traffic from the local clusters together into one place. This way clients can pick a local or a global view of the data, and we can have everyone still stay in their local Kafka clusters.
-Todd On Fri, Feb 27, 2015 at 4:21 PM, Yang <[email protected]> wrote: > we tested our new application that reads and writes to kafka. > > at first we found the access latency is very high. then we realized that > it's because the client and server are in different colos. moving them > together reduces down the access time to < 4 ms. > > > I was wondering if there are any techniques/properties to set , or maybe > enhancements to be made to the kafka code, that could cater to such > situations ? > > Thanks > Yang >
