This is another potential use-case for message metadata. i.e., if we had a DC/environment field in the header you could easily set up a two-way mirroring pipeline. The mirror-maker can just filter out messages that originated in the source cluster. For this to be efficient the mirror maker should run in the source cluster. (We traditionally run it in the target cluster but there is no hard and fast requirement that it be set up that way).
Thanks, Joel On Mon, Oct 20, 2014 at 07:43:26AM -0700, Neha Narkhede wrote: > Another way to set up this kind of mirroring is by deploying 2 clusters in > each DC - a local Kafka cluster and an aggregate Kafka cluster. The mirror > maker copies data from both the DC's local clusters into the aggregate > clusters. So if you want access to a topic with data from both DC's, you > subscribe to the aggregate cluster. > > Thanks, > Neha > > On Mon, Oct 20, 2014 at 7:07 AM, Erik van oosten < > e.vanoos...@grons.nl.invalid> wrote: > > > Hi, > > > > We have 2 data centers that produce events. Each DC has to process events > > from both DCs. > > > > I had the following in mind: > > > > DC 1 | DC 2 > > events | events > > + + + | + + + > > | | | | | | | > > v v v | v v v > > +----------------+ | +----------------+ > > | Receiver topic | | | Receiver topic | > > +----------------+ +----------------+ > > | | mirroring | | > > | | +------------------+ | > > | | | | > > | +--------------------+ | > > v v v v > > +----------------+ | +----------------+ > > | Consumer topic | | | Consumer topic | > > +----------------+ | +----------------+ > > + + + | + + + > > | | | | | | | > > v v v | v v v > > consumers | consumers > > > > > > As each DC has a single Kafka cluster, on each DC the receiver topic and > > consumer topic needs to be on the same cluster. > > Unfortunately, mirror maker does not seem to support mirroring to a topic > > with another name. > > > > Is there another tool we could use? > > Or, is there another approach for producing and consuming from 2 DCs? > > > > Kind regards, > > Erik. > > > > — > > Erik van Oosten > > http://www.day-to-day-stuff.blogspot.nl/ > > > >
