Hi Steven,
That doesn't work. In your proposal mirrormaker in once DC would copy
messages from topic A to the other DC in topic A. However, in the other
DC there is a mirrormaker which does the same, creating a loop. Messages
will be duplicated, triplicated, etc in a never ending loop.
Mirroring to another topic would work (mirrormaker doesn't support
that), and so would mirroring to another cluster. Neha's proposal would
work also but I assume its a lot more work for the Kafka internals and
therefor IMHO wouldn't meet the kiss principle.
Kind regards,
Erik.
Steven Wu schreef op 22-10-14 om 01:48:
I think it doesn't have to be two more clusters. can be just two more
topics. MirrorMaker can copy from source topics in both regions into one
aggregate topic.
On Tue, Oct 21, 2014 at 1:54 AM, Erik van oosten <
e.vanoos...@grons.nl.invalid> wrote:
Thanks Neha,
Unfortunately, the maintenance overhead of 2 more clusters is not
acceptable to us.
Would you accept a pull request on mirror maker that would rename topics
on the fly?
For example by accepting the parameter rename:
—rename src1/dest1,src2/dest2
or, extended with RE support:
—rename old_(.*)/new_\1
Kind regards,
Erik.
Op 20 okt. 2014, om 16:43 heeft Neha Narkhede <neha.narkh...@gmail.com>
het volgende geschreven:
Another way to set up this kind of mirroring is by deploying 2 clusters
in
each DC - a local Kafka cluster and an aggregate Kafka cluster. The
mirror
maker copies data from both the DC's local clusters into the aggregate
clusters. So if you want access to a topic with data from both DC's, you
subscribe to the aggregate cluster.
Thanks,
Neha
On Mon, Oct 20, 2014 at 7:07 AM, Erik van oosten <
e.vanoos...@grons.nl.invalid> wrote:
Hi,
We have 2 data centers that produce events. Each DC has to process
events
from both DCs.
I had the following in mind:
DC 1 | DC 2
events | events
+ + + | + + +
| | | | | | |
v v v | v v v
+----------------+ | +----------------+
| Receiver topic | | | Receiver topic |
+----------------+ +----------------+
| | mirroring | |
| | +------------------+ |
| | | |
| +--------------------+ |
v v v v
+----------------+ | +----------------+
| Consumer topic | | | Consumer topic |
+----------------+ | +----------------+
+ + + | + + +
| | | | | | |
v v v | v v v
consumers | consumers
As each DC has a single Kafka cluster, on each DC the receiver topic and
consumer topic needs to be on the same cluster.
Unfortunately, mirror maker does not seem to support mirroring to a
topic
with another name.
Is there another tool we could use?
Or, is there another approach for producing and consuming from 2 DCs?
Kind regards,
Erik.
—
Erik van Oosten
http://www.day-to-day-stuff.blogspot.nl/
--
Erik van Oosten
http://www.day-to-day-stuff.blogspot.com/
