[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931776#comment-16931776
]
Ryanne Dolan commented on KAFKA-7500:
-------------------------------------
[~qihong] That's all correct :)
In fact, KIP-382 mentions a "primary" cluster (in your case "C") and a single
Connect cluster being used to replicate data between any number of other
clusters. One way to do this is with a SinkConnector (part of the KIP and
coming later) s.t. records travel _through_ the primary cluster to other
clusters. I believe you could also override the Connect Worker's internal
Producer config to write directly to another cluster, s.t., as you say, state
is stored in one cluster but records go to another. I've never tried that,
ymmv, but I suspect it'd work as you've described.
Notice that, generally, you will end up with multiple Connect clusters anyway
-- one in each DC -- for performance and HA reasons. At that point you are back
to managing multiple Connectors on multiple Connect clusters. MM2's top-level
driver manages that complexity for you by automatically spinning up and
configuring all the required Workers.
re a REST API, the MM2 driver essentially turns off the REST service inside
each Herder. This is because the current Connect REST API doesn't support
having multiple Herders or Worker configs, so we'd need to sorta abuse the
Connect REST API to get it to work. However, there was much discussion around
KIP-382 re an MM2 REST API, and there are several good ideas floating around.
These were ultimately deferred, but not ruled out.
Also, fwiw, I have successfully run an MM2 cluster with the Connect REST API
turned on, with each Herder's endpoints wrapped in a higher-level API. I have
done this successfully with a reverse proxy as well as with a fork of Connect's
RestServer. This enables you to start/stop/configure individual connectors
within the MM2 cluster, if you so wish.
And finally: MM2 is very pluggable. For example, you can drop in a TopicFilter
that grabs dynamic whitelists from somewhere. I happen to know this works very
well :)
> MirrorMaker 2.0 (KIP-382)
> -------------------------
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
> Affects Versions: 2.4.0
> Reporter: Ryanne Dolan
> Assignee: Manikumar
> Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
