[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927863#comment-16927863
]
Qihong Chen edited comment on KAFKA-7500 at 9/11/19 6:13 PM:
-------------------------------------------------------------
[~ryannedolan] Thanks for all your efforts on MM2. Appreciate your input on the
following questions.
I'm trying to replicate topics from one cluster to another, include topic data
and related consumers' offset. But I only see the topic data was replicated,
not consumer offsets.
Here's my mm2.properties
{code:java}
clusters = dr1, dr2
dr1.bootstrap.servers = 10.1.0.4:9092,10.1.0.5:9092,10.1.0.6:9092
dr2.bootstrap.servers = 10.2.0.4:9092,10.2.0.5:9092,10.2.0.6:9092
# only allow replication dr1 -> dr2
dr1->dr2.enabled = true
dr1->dr2.topics = test.*
dr1->dr2.groups = test.*
dr1->dr2.emit.heartbeats.enabled = false
dr2->dr1.enabled = false
dr2->dr1.emit.heartbeats.enabled = false
{code}
Here's how I started MM2 cluster (dr2 as the nearby cluster)
{code:java}
nohup bin/connect-mirror-maker.sh mm2.properties --clusters dr2 > mm2.log 2>&1 &
{code}
On dr1, there's topic *test1*, and consumer group *test1grp* for topic test1.
On dr2, I found following topics
{code:java}
__consumer_offsets
dr1.checkpoints.internal
dr1.test1
heartbeats
mm2-configs.dr1.internal
mm2-offsets.dr1.internal
mm2-status.dr1.internal
{code}
But couldn't find any consumer groups on dr2 related to consumer group
*test1grp*.
Could you please let me know in detail how to migrate consumer group *test1grp*
from dr1 to dr2?, i.e. what command(s) need to run to set up the offset for
*test1grp* on dr2 before consume topic *dr1.test1* ?
By the way, how to set up and run this in a Kafka connect cluster? i.e., how to
set up
MirrorSourceConnector, MirrorCheckpointConnector in a connect cluster? Is
there document about this?
was (Author: qihong):
[~ryannedolan] Thanks for all your efforts on MM2. Appreciate your input on the
following questions.
I'm trying to replicate topics from one cluster to another, include topic data
and related consumers' offset. But I only see the topic data was replicated,
not consumer offsets.
Here's my mm2.properties
{code:java}
clusters = dr1, dr2
dr1.bootstrap.servers = 10.1.0.4:9092,10.1.0.5:9092,10.1.0.6:9092
dr2.bootstrap.servers = 10.2.0.4:9092,10.2.0.5:9092,10.2.0.6:9092
# only allow replication dr1 -> dr2
dr1->dr2.enabled = true
dr1->dr2.topics = test.*
dr1->dr2.groups = test.*
dr1->dr2.emit.heartbeats.enabled = false
dr2->dr1.enabled = false
dr2->dr1.emit.heartbeats.enabled = false
{code}
Here's how I started MM2 cluster (dr2 as the nearby cluster)
{code:java}
nohup bin/connect-mirror-maker.sh mm2.properties --clusters dr2 > mm2.log 2>&1 &
{code}
On dr1, there's topic *test1*, and consumer group *test1grp* for topic test1.
On dr2, I found following topics
{code:java}
__consumer_offsets
dr1.checkpoints.internal
dr1.test1
heartbeats
mm2-configs.dr1.internal
mm2-offsets.dr1.internal
mm2-status.dr1.internal
{code}
But couldn't find any consumer groups on dr2 related consumer group *test1grp*.
Could you please let me know in detail how to migrate consumer group
*test1grp* from dr1 to dr2?, i.e. what command(s) need to run to set up the
offset for *test1grp* on dr2 before consume topic *dr1.test1* ?
By the way, how to set up and run this in a Kafka connect cluster? i.e., how to
set up
MirrorSourceConnector, MirrorCheckpointConnector in a connect cluster? Is
there document about this?
> MirrorMaker 2.0 (KIP-382)
> -------------------------
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
> Affects Versions: 2.4.0
> Reporter: Ryanne Dolan
> Assignee: Manikumar
> Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
