I think it doesn't have to be two more clusters. can be just two more topics. MirrorMaker can copy from source topics in both regions into one aggregate topic.
On Tue, Oct 21, 2014 at 1:54 AM, Erik van oosten < e.vanoos...@grons.nl.invalid> wrote: > Thanks Neha, > > Unfortunately, the maintenance overhead of 2 more clusters is not > acceptable to us. > > Would you accept a pull request on mirror maker that would rename topics > on the fly? > > For example by accepting the parameter rename: > —rename src1/dest1,src2/dest2 > or, extended with RE support: > —rename old_(.*)/new_\1 > > Kind regards, > Erik. > > > Op 20 okt. 2014, om 16:43 heeft Neha Narkhede <neha.narkh...@gmail.com> > het volgende geschreven: > > > Another way to set up this kind of mirroring is by deploying 2 clusters > in > > each DC - a local Kafka cluster and an aggregate Kafka cluster. The > mirror > > maker copies data from both the DC's local clusters into the aggregate > > clusters. So if you want access to a topic with data from both DC's, you > > subscribe to the aggregate cluster. > > > > Thanks, > > Neha > > > > On Mon, Oct 20, 2014 at 7:07 AM, Erik van oosten < > > e.vanoos...@grons.nl.invalid> wrote: > > > >> Hi, > >> > >> We have 2 data centers that produce events. Each DC has to process > events > >> from both DCs. > >> > >> I had the following in mind: > >> > >> DC 1 | DC 2 > >> events | events > >> + + + | + + + > >> | | | | | | | > >> v v v | v v v > >> +----------------+ | +----------------+ > >> | Receiver topic | | | Receiver topic | > >> +----------------+ +----------------+ > >> | | mirroring | | > >> | | +------------------+ | > >> | | | | > >> | +--------------------+ | > >> v v v v > >> +----------------+ | +----------------+ > >> | Consumer topic | | | Consumer topic | > >> +----------------+ | +----------------+ > >> + + + | + + + > >> | | | | | | | > >> v v v | v v v > >> consumers | consumers > >> > >> > >> As each DC has a single Kafka cluster, on each DC the receiver topic and > >> consumer topic needs to be on the same cluster. > >> Unfortunately, mirror maker does not seem to support mirroring to a > topic > >> with another name. > >> > >> Is there another tool we could use? > >> Or, is there another approach for producing and consuming from 2 DCs? > >> > >> Kind regards, > >> Erik. > >> > >> — > >> Erik van Oosten > >> http://www.day-to-day-stuff.blogspot.nl/ > >> > >> > >
