Hello and thank you for the reply! My problem is not with consumption of messages, because as you said, MirrorMaker2 knows how to deal with the consumer offsets. Rather my problem is with source connectors and the topic connect-offsets.
Because Kafka connect manages where it stopped reading from a source using to topic, the second connect cluster is not aware of the offsets and I can get duplicates. I know MirrorMaker2 does not guarantee exactly-once delivery but is there a way to maybe sync to topics so the kafka connect will be aware of each other offset. Daniel > On 19 Mar 2021, at 9:09, Ning Zhang <ning2008w...@gmail.com> wrote: > Hi Daniel, MirrorMaker2 creates its own "offsets" topic to track the process > of consumption. > > just my 2 cents - If you already have two Kafka connect clusters in two > different sites, it sounds practical to: > (1) use "cluster" mode, instead of "dedicated" mode of MirrorMaker2 > (2) add one "MirrorMaker" connector on each Kafka connect cluster and do > "one-way" replication. As you have two "MirrorMaker" connectors, it should > behave like "active-active" deployment. > > Just in case you only have one Kafka connect cluster, do "active-active" > replication, rather than "one-way" > > On 2021/03/18 08:15:40, Daniel Beilin <dandanie...@gmail.com> wrote: >> Hi everyone, >> I'm trying to create an active-active deployment of a kafka cluster between >> two data centers using MirrorMaker2, but I'm facing a problem. >> In my deployment I have Kafka Connect in both sites which each of them >> connect to different database using sink and source connectors (MongoDB >> source connector , JDBC sink/source connector) >> I’d like to know what’s the best practice for active-active is using Kafka >> connect , since I noticed the “connect-offsets” topic is not replicated in >> mm2. >> >> Best regards, >> Daniel
