Erik, I don't know that mirrormaker can't write to a different topic. but it might be an useful feature request to mirrormaker.
On Wed, Oct 22, 2014 at 12:21 AM, Erik van Oosten < e.vanoos...@grons.nl.invalid> wrote: > Hi Steven, > > That doesn't work. In your proposal mirrormaker in once DC would copy > messages from topic A to the other DC in topic A. However, in the other DC > there is a mirrormaker which does the same, creating a loop. Messages will > be duplicated, triplicated, etc in a never ending loop. > > Mirroring to another topic would work (mirrormaker doesn't support that), > and so would mirroring to another cluster. Neha's proposal would work also > but I assume its a lot more work for the Kafka internals and therefor IMHO > wouldn't meet the kiss principle. > > Kind regards, > Erik. > > > Steven Wu schreef op 22-10-14 om 01:48: > > I think it doesn't have to be two more clusters. can be just two more >> topics. MirrorMaker can copy from source topics in both regions into one >> aggregate topic. >> >> On Tue, Oct 21, 2014 at 1:54 AM, Erik van oosten < >> e.vanoos...@grons.nl.invalid> wrote: >> >> Thanks Neha, >>> >>> Unfortunately, the maintenance overhead of 2 more clusters is not >>> acceptable to us. >>> >>> Would you accept a pull request on mirror maker that would rename topics >>> on the fly? >>> >>> For example by accepting the parameter rename: >>> —rename src1/dest1,src2/dest2 >>> or, extended with RE support: >>> —rename old_(.*)/new_\1 >>> >>> Kind regards, >>> Erik. >>> >>> >>> Op 20 okt. 2014, om 16:43 heeft Neha Narkhede <neha.narkh...@gmail.com> >>> het volgende geschreven: >>> >>> Another way to set up this kind of mirroring is by deploying 2 clusters >>>> >>> in >>> >>>> each DC - a local Kafka cluster and an aggregate Kafka cluster. The >>>> >>> mirror >>> >>>> maker copies data from both the DC's local clusters into the aggregate >>>> clusters. So if you want access to a topic with data from both DC's, you >>>> subscribe to the aggregate cluster. >>>> >>>> Thanks, >>>> Neha >>>> >>>> On Mon, Oct 20, 2014 at 7:07 AM, Erik van oosten < >>>> e.vanoos...@grons.nl.invalid> wrote: >>>> >>>> Hi, >>>>> >>>>> We have 2 data centers that produce events. Each DC has to process >>>>> >>>> events >>> >>>> from both DCs. >>>>> >>>>> I had the following in mind: >>>>> >>>>> DC 1 | DC 2 >>>>> events | events >>>>> + + + | + + + >>>>> | | | | | | | >>>>> v v v | v v v >>>>> +----------------+ | +----------------+ >>>>> | Receiver topic | | | Receiver topic | >>>>> +----------------+ +----------------+ >>>>> | | mirroring | | >>>>> | | +------------------+ | >>>>> | | | | >>>>> | +--------------------+ | >>>>> v v v v >>>>> +----------------+ | +----------------+ >>>>> | Consumer topic | | | Consumer topic | >>>>> +----------------+ | +----------------+ >>>>> + + + | + + + >>>>> | | | | | | | >>>>> v v v | v v v >>>>> consumers | consumers >>>>> >>>>> >>>>> As each DC has a single Kafka cluster, on each DC the receiver topic >>>>> and >>>>> consumer topic needs to be on the same cluster. >>>>> Unfortunately, mirror maker does not seem to support mirroring to a >>>>> >>>> topic >>> >>>> with another name. >>>>> >>>>> Is there another tool we could use? >>>>> Or, is there another approach for producing and consuming from 2 DCs? >>>>> >>>>> Kind regards, >>>>> Erik. >>>>> >>>>> — >>>>> Erik van Oosten >>>>> http://www.day-to-day-stuff.blogspot.nl/ >>>>> >>>>> >>>>> >>> > > -- > Erik van Oosten > http://www.day-to-day-stuff.blogspot.com/ > >
