[
https://issues.apache.org/jira/browse/KAFKA-12150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259373#comment-17259373
]
Ara Zarifian edited comment on KAFKA-12150 at 1/6/21, 3:12 AM:
---------------------------------------------------------------
*Offset replication scenario with 1 MirrorMaker replica:*
Two new clusters:
{code}
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER1 --list
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER2 --list
{code}
Spin up MirrorMaker and create/list topics:
{code}
[root@localhost bin]# kubectl scale deploy -n azure1-dev
kmm-replication1-replication2-ks7sklq9 --replicas=1 # (from 0)
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER1 --create
--topic foo --replication-factor 3 --partitions 15
Created topic foo.
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER2 --create
--topic bar --replication-factor 3 --partitions 15
Created topic bar.
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER1 --list
__consumer_offsets
foo
heartbeats
mm2-configs.westus.internal
mm2-offset-syncs.westus.internal
mm2-offsets.westus.internal
mm2-status.westus.internal
westus.bar
westus.checkpoints.internal
westus.heartbeats
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER2 --list
__consumer_offsets
bar
eastus2.checkpoints.internal
eastus2.foo
eastus2.heartbeats
heartbeats
mm2-configs.eastus2.internal
mm2-offset-syncs.eastus2.internal
mm2-offsets.eastus2.internal
mm2-status.eastus2.internal
{code}
Produce to cluster 1:
{code}
[root@localhost bin]# ./kafka-console-producer.sh --bootstrap-server $CLUSTER1
--topic foo
>foo
>bar
>baz
>^C[root@localhost bin]#
{code}
Consume from topic in cluster 1 and check offset information:
{code}
[root@localhost bin]# ./kafka-console-consumer.sh --bootstrap-server $CLUSTER1
--topic foo --from-beginning --consumer-property group.id=test
foo
bar
baz
^CProcessed a total of 3 messages
[root@localhost bin]# ./kafka-consumer-groups.sh --bootstrap-server $CLUSTER1
--describe --group test
Consumer group 'test' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID
test foo 6 1 1 0
- - -
test foo 7 0 0 0
- - -
test foo 4 0 0 0
- - -
test foo 5 1 1 0
- - -
test foo 2 0 0 0
- - -
test foo 3 0 0 0
- - -
test foo 0 0 0 0
- - -
test foo 1 0 0 0
- - -
test foo 14 0 0 0
- - -
test foo 12 0 0 0
- - -
test foo 13 0 0 0
- - -
test foo 10 0 0 0
- - -
test foo 11 0 0 0
- - -
test foo 8 1 1 0
- - -
test foo 9 0 0 0
- - -
{code}
Check offset information in cluster 2:
{code}
[root@localhost bin]# ./kafka-consumer-groups.sh --bootstrap-server $CLUSTER2
--describe --group test
Consumer group 'test' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID
test eastus2.foo 14 0 0 0
- - -
test eastus2.foo 13 0 0 0
- - -
test eastus2.foo 10 0 0 0
- - -
test eastus2.foo 9 0 0 0
- - -
test eastus2.foo 12 0 0 0
- - -
test eastus2.foo 11 0 0 0
- - -
test eastus2.foo 6 1 1 0
- - -
test eastus2.foo 5 1 1 0
- - -
test eastus2.foo 8 1 1 0
- - -
test eastus2.foo 7 0 0 0
- - -
test eastus2.foo 2 0 0 0
- - -
test eastus2.foo 1 0 0 0
- - -
test eastus2.foo 4 0 0 0
- - -
test eastus2.foo 3 0 0 0
- - -
test eastus2.foo 0 0 0 0
- - -
{code}
With 1 MirrorMaker replica, I was able to replicate the above result several
times.
*Offset replication scenario with 3 MirrorMaker replica:*
If, however, the MirrorMaker deployment is deployed from scratch with 3
replicas against 2 new clusters, the consumer group itself doesn't synced
across clusters:
{code}
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER1 --list
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER2 --list
[root@localhost bin]# kubectl scale deploy -n azure1-dev
kmm-replication1-replication2-ks7sklq9 --replicas=3 # (from 0)
{code}
<same set of steps above>
{code}
[root@localhost bin]# ./kafka-consumer-groups.sh --bootstrap-server $CLUSTER2
--describe --group test
Error: Consumer group 'test' does not exist.
{code}
was (Author: arzarif):
*Offset replication scenario with 1 MirrorMaker replica:*
Two new clusters:
{code}
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER1 --list
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER2 --list
{code}
Spin up MirrorMaker and create/list topics:
{code}
[root@localhost bin]# kubectl scale deploy -n azure1-dev
kmm-replication1-replication2-ks7sklq9 --replicas=1
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER1 --create
--topic foo --replication-factor 3 --partitions 15
Created topic foo.
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER2 --create
--topic bar --replication-factor 3 --partitions 15
Created topic bar.
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER1 --list
__consumer_offsets
foo
heartbeats
mm2-configs.westus.internal
mm2-offset-syncs.westus.internal
mm2-offsets.westus.internal
mm2-status.westus.internal
westus.bar
westus.checkpoints.internal
westus.heartbeats
[root@localhost bin]# ./kafka-topics.sh --bootstrap-server $CLUSTER2 --list
__consumer_offsets
bar
eastus2.checkpoints.internal
eastus2.foo
eastus2.heartbeats
heartbeats
mm2-configs.eastus2.internal
mm2-offset-syncs.eastus2.internal
mm2-offsets.eastus2.internal
mm2-status.eastus2.internal
{code}
Produce to cluster 1:
{code}
[root@localhost bin]# ./kafka-console-producer.sh --bootstrap-server $CLUSTER1
--topic foo
>foo
>bar
>baz
>^C[root@localhost bin]#
{code}
Consume from topic in cluster 1 and check offset information:
{code}
[root@localhost bin]# ./kafka-console-consumer.sh --bootstrap-server $CLUSTER1
--topic foo --from-beginning --consumer-property group.id=test
foo
bar
baz
^CProcessed a total of 3 messages
[root@localhost bin]# ./kafka-consumer-groups.sh --bootstrap-server $CLUSTER1
--describe --group test
Consumer group 'test' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID
test foo 6 1 1 0
- - -
test foo 7 0 0 0
- - -
test foo 4 0 0 0
- - -
test foo 5 1 1 0
- - -
test foo 2 0 0 0
- - -
test foo 3 0 0 0
- - -
test foo 0 0 0 0
- - -
test foo 1 0 0 0
- - -
test foo 14 0 0 0
- - -
test foo 12 0 0 0
- - -
test foo 13 0 0 0
- - -
test foo 10 0 0 0
- - -
test foo 11 0 0 0
- - -
test foo 8 1 1 0
- - -
test foo 9 0 0 0
- - -
{code}
Check offset information in cluster 2:
{code}
[root@localhost bin]# ./kafka-consumer-groups.sh --bootstrap-server $CLUSTER2
--describe --group test
Consumer group 'test' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST CLIENT-ID
test eastus2.foo 14 0 0 0
- - -
test eastus2.foo 13 0 0 0
- - -
test eastus2.foo 10 0 0 0
- - -
test eastus2.foo 9 0 0 0
- - -
test eastus2.foo 12 0 0 0
- - -
test eastus2.foo 11 0 0 0
- - -
test eastus2.foo 6 1 1 0
- - -
test eastus2.foo 5 1 1 0
- - -
test eastus2.foo 8 1 1 0
- - -
test eastus2.foo 7 0 0 0
- - -
test eastus2.foo 2 0 0 0
- - -
test eastus2.foo 1 0 0 0
- - -
test eastus2.foo 4 0 0 0
- - -
test eastus2.foo 3 0 0 0
- - -
test eastus2.foo 0 0 0 0
- - -
{code}
With 1 MirrorMaker replica, I was able to replicate the above result several
times.
*Offset replication scenario with 3 MirrorMaker replica:*
If, however, the MirrorMaker deployment is scaled to 3 replicas, the consumer
group itself isn't consistently synced across clusters:
{code}
[root@localhost bin]# kubectl scale deploy -n azure1-dev
kmm-replication1-replication2-ks7sklq9 --replicas=3
{code}
<same set of steps above>
{code}
[root@localhost bin]# ./kafka-consumer-groups.sh --bootstrap-server $CLUSTER2
--describe --group test
Error: Consumer group 'test' does not exist.
{code}
> Consumer group refresh not working with clustered MM2 setup
> -----------------------------------------------------------
>
> Key: KAFKA-12150
> URL: https://issues.apache.org/jira/browse/KAFKA-12150
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.7.0
> Reporter: Ara Zarifian
> Priority: Major
>
> I'm running MM2 with Kafka 2.7 with the following configuration:
> {code}
> clusters = eastus2, westus
> eastus2.bootstrap.servers = clusrter1.example.com:9092
> westus.bootstrap.servers = cluster2.example.com:9092
> eastus2->westus.enabled = true
> eastus2->westus.topics = .*
> westus->eastus2.enabled = true
> westus->eastus2.topics = .*
> refresh.topics.enabled = true
> refresh.topics.interval.seconds = 5
> refresh.groups.enabled = true
> refresh.groups.interval.seconds = 5
> sync.topic.configs.enabled = true
> sync.topic.configs.interval.seconds = 5
> sync.topic.acls.enabled = false
> sync.topic.acls.interval.seconds = 5
> sync.group.offsets.enabled = true
> sync.group.offsets.interval.seconds = 5
> emit.checkpoints.enabled = true
> emit.checkpoints.interval.seconds = 5
> emit.heartbeats.enabled = true
> emit.heartbeats.interval.seconds = 5
> replication.factor = 3
> checkpoints.topic.replication.factor = 3
> heartbeats.topic.replication.factor = 3
> offset-syncs.topic.replication.factor = 3
> offset.storage.replication.factor = 3
> status.storage.replication.factor = 3
> config.storage.replication.factor = 3
> {code}
> More specifically, I'm running multiple instances of MM2 with the above
> configuration within Kubernetes pods. I was testing the new automatic
> consumer group offset translation functionality and noticed what appears to
> be a problem when running more than 1 instance of MM2 in this fashion.
> Based on [on the
> KEP|https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0],
> I should be able to run multiple instances in this manner (see "Running a
> dedicated MirrorMaker cluster"), however, I noticed that when enabling
> replication using a 3-instance MM2 cluster, consumer groups were not
> synchronizing across clusters at all.
> When running through my test case with a single MM2 instance, consumer group
> synchronization appears to work as expected consistently. When running
> through my 3-node test case, synchronization begins as soon as I scale the
> number of replicas to 1.
> Am I misinterpreting the manner in which the KEP describes MM2 clusters or is
> this interaction an unexpected one?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
