Hi Folks, I'm running mirror maker as a dedicated cluster as given in the mirrormaker 2 doc. but for some reason when I add new topics and deploy the mirror maker it's not detecting the new topics at all, even the config dumps in the mirror maker startup logs don't show the newly added topics.
I've attached the config that I'm using, initially I assumed that there might be some refresh configuration option either in connect or mirror maker, but the connect rest api doesn't seem to be working in this mode and also couldn't find any refresh configuration option. Any ideas on this? Thank you in advance ``` clusters = src-cluster, dst-cluster # disable topic prefixes src-cluster.replication.policy.separator = dst-cluster.replication.policy.separator = replication.policy.separator = source.cluster.alias = target.cluster.alias = # enable idemptotence source.cluster.producer.enable.idempotence = true target.cluster.producer.enable.idempotence = true # connection information for each cluster # This is a comma separated host:port pairs for each cluster # for e.g. "A_host1:9092, A_host2:9092, A_host3:9092" src-cluster.bootstrap.servers = sng-kfnode1.internal:9092,sng-kfnode1.internal:9092,sng-kfnode1.internal:9092 dst-cluster.bootstrap.servers = prod-online-v2-kafka-1.internal:9092,prod-online-v2-kafka-2.internal:9092,prod-online-v2-kafka-3.internal:9092,prod-online-v2-kafka-4.internal:9092,prod-online-v2-kafka-5.internal:9092 # regex which defines which topics gets replicated. For eg "foo-.*" src-cluster->dst-cluster.topics = topic1,topic2 # client-id src-cluster.client.id = prod-mm2-onlinev1-to-onlinev2-consumer-v0 dst-cluster.client.id = prod-mm2-onlinev1-to-onlinev2-producer-v0 # group.instance.id=_mirror_make_instance_1 # consumer should periodically emit heartbeats src-cluster->dst-cluster.consumer.auto.offset.reset = earliest src-cluster->dst-cluster.consumer.overrides.auto.offset.reset = earliest # connector should periodically emit heartbeats src-cluster->dst-cluster.emit.heartbeats.enabled = true # frequency of heartbeats, default is 5 seconds src-cluster->dst-cluster.emit.heartbeats.interval.seconds = 10 # connector should periodically emit consumer offset information src-cluster->dst-cluster.emit.checkpoints.enabled = true # frequency of checkpoints, default is 5 seconds src-cluster->dst-cluster.emit.checkpoints.interval.seconds = 10 # whether to monitor source cluster ACLs for changes src-cluster->dst-cluster.sync.topic.acls.enabled = false # whether or not to monitor source cluster for configuration changes src-cluster->dst-cluster.sync.topic.configs.enabled = true # add retention.ms to the default list given in the DefaultConfigPropertyFilter # https://github.com/apache/kafka/blob/889fd31b207b86db6d059792131d14389639d9e4/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/DefaultConfigPropertyFilter.java#L33-L38 src-cluster->dst-cluster.config.properties.blacklist = follower\\.replication\\.throttled\\.replicas, \ leader\\.replication\\.throttled\\.replicas, \ message\\.timestamp\\.difference\\.max\\.ms, \ message\\.timestamp\\.type, \ unclean\\.leader\\.election\\.enable, \ min\\.insync\\.replicas, \ retention\\.ms # connector should periodically check for new topics src-cluster->dst-cluster.refresh.topics.enabled = true # frequency to check source cluster for new topics, default is 5 seconds src-cluster->dst-cluster.refresh.topics.interval.seconds = 300 # enable and configure individual replication flows src-cluster->dst-cluster.enabled = true dst-cluster->src-cluster.enabled = false # Setting replication factor of newly created remote topics # replication.factor=2 ############################# Internal Topic Settings ############################# # The replication factor for mm2 internal topics "heartbeats", "B.checkpoints.internal" and # "mm2-offset-syncs.B.internal" # For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3. checkpoints.topic.replication.factor=3 # 14 days checkpoints.topic.retention.ms=1209600000 heartbeats.topic.replication.factor=3 offset-syncs.topic.replication.factor=3 # The replication factor for connect internal topics "mm2-configs.B.internal", "mm2-offsets.B.internal" and # "mm2-status.B.internal" # For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3. offset.storage.replication.factor=3 status.storage.replication.factor=3 config.storage.replication.factor=3 # customize as needed # replication.policy.separator = _ # sync.topic.acls.enabled = false # emit.heartbeats.interval.seconds = 5 ``` Thanks
