Hi Steven,
That doesn't work. In your proposal mirrormaker in once DC would copy
messages from topic A to the other DC in topic A. However, in the other
DC there is a mirrormaker which does the same, creating a loop. Messages
will be duplicated, triplicated, etc in a never ending loop.
Erik, I don't know that mirrormaker can't write to a different topic. but
it might be an useful feature request to mirrormaker.
On Wed, Oct 22, 2014 at 12:21 AM, Erik van Oosten
e.vanoos...@grons.nl.invalid wrote:
Hi Steven,
That doesn't work. In your proposal mirrormaker in once DC would
Thanks Neha,
Unfortunately, the maintenance overhead of 2 more clusters is not acceptable to
us.
Would you accept a pull request on mirror maker that would rename topics on the
fly?
For example by accepting the parameter rename:
—rename src1/dest1,src2/dest2
or, extended with RE support:
I think it doesn't have to be two more clusters. can be just two more
topics. MirrorMaker can copy from source topics in both regions into one
aggregate topic.
On Tue, Oct 21, 2014 at 1:54 AM, Erik van oosten
e.vanoos...@grons.nl.invalid wrote:
Thanks Neha,
Unfortunately, the maintenance
Hi,
We have 2 data centers that produce events. Each DC has to process events from
both DCs.
I had the following in mind:
DC 1 | DC 2
events |events
+ + + | + + +
| | | | | | |
v
Another way to set up this kind of mirroring is by deploying 2 clusters in
each DC - a local Kafka cluster and an aggregate Kafka cluster. The mirror
maker copies data from both the DC's local clusters into the aggregate
clusters. So if you want access to a topic with data from both DC's, you
This is another potential use-case for message metadata. i.e., if we
had a DC/environment field in the header you could easily set up a
two-way mirroring pipeline. The mirror-maker can just filter out
messages that originated in the source cluster. For this to be
efficient the mirror maker should
