Hey Jeroen,
Thanks for looking into the collision theory!
> There are no offset collisions as the offset-syncs albeit being stored on
> the same cluster, offsets from A->B are stored
> in mm2-offset-syncs.b.internal whereas offsets from B->A are stored
> in mm2-offset-syncs.a.internal.
The
Ryanne,
> > b.a.replicate-me-0
> That's actually impossible with MM2.
Thanks, I see the isCycle check in MirrorSourceConnector. That makes
me even more curious how the renameTopicPartition method triggers
without a change such as the one that Jeroen has prototyped, since the
only thing that
Hey Greg,
There are no offset collisions as the offset-syncs albeit being stored on
the same cluster, offsets from A->B are stored
in mm2-offset-syncs.b.internal whereas offsets from B->A are stored
in mm2-offset-syncs.a.internal.
What's curious though is the B->A checkpoint connector (which has
> b.a.replicate-me-0
That's actually impossible with MM2. It won't allow such topics. It would
be pointless to replicate data from A back to A, or from B back to B. The
data is already there.
MM2's replication logic is significantly more advanced than just adding a
prefix. Tho it does appear
Hey Jeroen,
Thanks for sharing your prototype! It is very interesting!
> I couldn't reproduce your hypothesis.
I think my hypothesis was for another setup which didn't involve code
changes, and instead relied on A->B->A round trip replication to
produce the "backwards" offset syncs.
I believe
I see, makes complete sense to me. I built a custom version [1] based off
of Kafka 3.5.1 with bidirectional offset replication enabled so I could do
some more testing. Offset translation back upstream works well; I think
because of the reason Ryanne pointed out, both topics contain identical
data.
Hey Ryanne,
Thanks for the context, but I still don't see the situation where this
function is helpful.
Also "A's topic1 and B's a.topic1 should be the same data (minus
replication lag)." isn't true in the presence of failures/hard
restarts, compaction, and transaction markers.
Thanks,
Greg
On
> do you recall the purpose of [...] renameTopicPartition [?]
A's topic1 and B's a.topic1 should be the same data (minus replication
lag). You can't consume a record in a.topic1 that hasn't been replicated
yet -- a remote topic by definition does not have any records that MM2
didn't put there. So
Hi Jeroen,
I'm glad you're experimenting with MM2, and I hope we can give you
some more context to explain what you're seeing.
> I wrote a small program to produce these offset syncs for the prefixed
> topic, and this successfully triggers the Checkpoint connector to start
> replicating the
Thank you both for your swift responses!
Ryanne, the MirrorConnectorsIntegrationBaseTest only tests offset
replication in cases where the producer migrated to the secondary cluster
as well, starts feeding messages into the non-prefixed topic which are
subsequently consumed by the consumer on the
Jeroen, MirrorClient will correctly translate offsets for both failover and
failback, exactly as you describe. It's possible to automate failover and
failback using that logic. The integration tests automatically fail over
and fail back, for example. I've seen it done two ways: during startup
Hi Jeroen,
Thanks for looking into MM2 for your use-case!
I believe the "active-active" model that MM2 uses is really more of a
pair of opposing "active-passive" replication flows, and this extends
to the offset sync mechanism too.
MM2 doesn't change the state of the source topic or consumer
Hi all,
I'm exploring using the MirrorSourceConnector and MirrorCheckpointConnector
on Kafka Connect to setup active/active replication between two Kafka
clusters. Using the DefaultReplicationPolicy replication policy class,
messages originating from the source cluster get replicated as expected
13 matches
Mail list logo
