Hi,
After going through this https://issues.apache.org/jira/browse/KAFKA-10582 , I learnt that this is a similar behaviour seen with Mirrormaker2 and I found some success initially by restarting the kafka node in cluster B where mm2 instance was running. Initially the size of the topic increased to 167GB and now constantly reducing in every minute (size as of now is 39GB). I am stuck in moving ahead with the same and would really appreciate your help on the same. Thanks, Rijo Roy On Thursday, 2 December, 2021, 11:20:29 pm IST, Rijo Roy <[email protected]> wrote: Hi, Hope you all are safe and well! I found success in using Mirrormaker2 to sync data from source A (1 broker in a 3node cluster) to target B(1 broker in a 3node cluster) where I was running mirrormaker2 in the controller node in cluster B. Now, I see an issue with the sync for a topic whose size in cluster A is approximately 332GB but the same topic in cluster B is hardly 8KB. And this issues is not seen for any other topics as they are getting sync'ed as expected.. Could you please help me understand why this topic is failing to get sync'ed Source cluster A ============= kafka-topics --zookeeper zknode:2181 --describe --topic bigtopic Topic: bigtopic PartitionCount: 1 ReplicationFactor: 3 Configs: cleanup.policy=delete Topic: bigtopic Partition: 0 Leader: 1002 Replicas: 1002,1001,1003 Isr: 1003,1002,1001 du -sh /kafkadata/logs/bigtopic-0/ 331G /kafkadata/logs/bigtopic-0/ Target cluster B ============ kafka-topics --zookeeper zknodeB:2181 --describe --topic bigtopic Topic: bigtopic PartitionCount: 1 ReplicationFactor: 3 Configs: cleanup.policy=delete Topic: bigtopic Partition: 0 Leader: 6 Replicas: 6,4,5 Isr: 6,4,5 du -sh /kafkadata/logs/bigtopic-0/ 8.0K /kafkadata/logs/bigtopic-0/ And here is the mm2.properties that I use- # mm2.properties clusters = A, B A.bootstrap.servers = Ahostname:9092 B.bootstrap.servers = Bhostname:9092 A->B.enabled = true A->B.topics = somanytopicnames_including_bigtopic replication.factor=3 refresh.topics.interval.seconds = 10 refresh.groups.interval.seconds = 10 sync.topic.config.enabled = true replication.policy.separator = source.cluster.alias = target.cluster.alias = #replication.policy.class= io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy groups = .* tasks.max=100 emit.checkpoints.interval.seconds = 10 security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="CONFIDENTIAL" \ password="vvvvvvv"; Thanks & Regards, Rijo Roy
