[
https://issues.apache.org/jira/browse/KAFKA-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516755#comment-17516755
]
David Bros edited comment on KAFKA-13796 at 4/4/22 11:49 AM:
-------------------------------------------------------------
We solved the issue for now by not matching in the whitelist property, but this
is not a very good solution:
topics = ^(?!.\*(mm2_test_0|mm2_test_1)).\*
was (Author: JIRAUSER284538):
We solved the issue for now by not matching in the whitelist property, but this
is not a very good solution:
^(?!.\*(mm2_test_0|mm2_test_1)).\*
> MM2 - Topics Exclude/Blacklist not working
> ------------------------------------------
>
> Key: KAFKA-13796
> URL: https://issues.apache.org/jira/browse/KAFKA-13796
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.0.0
> Reporter: David Bros
> Priority: Blocker
> Fix For: 3.0.0
>
>
> Hi all,
> We are testing Kafka 3.0.0 in a development environment with Java 17 in a
> Centos 7 machine.
> When setting up MM2 replication between two clusters we exclude some topics
> we don't want MM2 to replicate via topics.exclude (we have also tried
> topics.blacklist)
> Even though the configuration is parsed correctly, we still get those topics
> replicated among clusters.
> Here is our config for the us_central cluster:
> ```
> # Clusters and bootstrap servers
> clusters=eu_west, us_central
> eu_west.bootstrap.servers=XXnet:9092,XX:9092
> us_central.bootstrap.servers=XX:9092
> us_central.consumer.auto.offset.reset=latest
> us_central.consumer.fetch.max.bytes=31457280
> us_central.consumer.fetch.max.wait.ms=10000
> us_central.consumer.max.poll.records=3000
> us_central.consumer.request.timeout.ms=60000
> eu_west.consumer.auto.offset.reset=latest
> eu_west.consumer.fetch.max.bytes=52428800
> eu_west.consumer.fetch.max.wait.ms=10000
> eu_west.consumer.max.poll.records=3000
> eu_west.consumer.request.timeout.ms=60000
> # Custom producer settings
> us_central.producer.max.request.size=27262976
> us_central.producer.batch.size=22000
> us_central.producer.compression.type=none
> us_central.producer.send.buffer.bytes=26214400
> us_central.producer.receive.buffer.bytes=26214400
> eu_west.producer.max.request.size=27262976
> eu_west.producer.batch.size=22000
> eu_west.producer.compression.type=none
> eu_west.producer.send.buffer.bytes=26214400
> eu_west.producer.receive.buffer.bytes=26214400
> # Topics configuration
> topics=.*
> topics.blacklist="mm2_test_0"
> # Tasks (threads)
> tasks.max=10
> # Groups
> groups=phx_netflow
> # Replication factors, these are 1 for lab
> replication.factor=1
> config.storage.replication.factor=1
> offset.storage.replication.factor=1
> status.storage.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> sync.group.offsets.replication.factor=1
> ## Config
> config.properties.exclude=local.retention.ms
> # Refresh rates
> us_central.refresh.topics.interval.seconds=15
> us_central.refresh.groups.interval.seconds=15
> eu_west.refresh.topics.interval.seconds=15
> eu_west.refresh.groups.interval.seconds=15
> ## Sync options
> sync.topic.acls.enabled=true
> offset-syncs.topic.replication.enabled=true
> sync.group.offsets.enabled=true
> # EU pulls us, syd, nyc
> us_central->eu_west.enabled=true
> us_central->eu_west.sync.group.offsets.enabled=false
> # US pulls eu, syd, nyc
> eu_west->us_central.enabled=true
> eu_west->us_central.sync.group.offsets.enabled=true
> ```
> Here is the configuration for eu_west cluster:
> ```
> # Clusters and bootstrap servers
> clusters=eu_west, us_central
> eu_west.bootstrap.servers=XX,XX:9092
> us_central.bootstrap.servers=XX:9092
> us_central.consumer.auto.offset.reset=latest
> us_central.consumer.fetch.max.bytes=31457280
> us_central.consumer.fetch.max.wait.ms=10000
> us_central.consumer.max.poll.records=3000
> us_central.consumer.request.timeout.ms=60000
> eu_west.consumer.auto.offset.reset=latest
> eu_west.consumer.fetch.max.bytes=52428800
> eu_west.consumer.fetch.max.wait.ms=10000
> eu_west.consumer.max.poll.records=3000
> eu_west.consumer.request.timeout.ms=60000
> # Custom producer settings
> us_central.producer.max.request.size=27262976
> us_central.producer.batch.size=22000
> us_central.producer.compression.type=none
> us_central.producer.send.buffer.bytes=26214400
> us_central.producer.receive.buffer.bytes=26214400
> eu_west.producer.max.request.size=27262976
> eu_west.producer.batch.size=22000
> eu_west.producer.compression.type=none
> eu_west.producer.send.buffer.bytes=26214400
> eu_west.producer.receive.buffer.bytes=26214400
> # Topics configuration
> topics=.*
> topics.blacklist="mm2_test_0"
> # Tasks (threads)
> tasks.max=10
> # Groups
> groups=ams_netflow
> # Replication factors, these are 1 for lab
> replication.factor=1
> config.storage.replication.factor=1
> offset.storage.replication.factor=1
> status.storage.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> sync.group.offsets.replication.factor=1
> ## Config
> config.properties.exclude=local.retention.ms
> # Refresh rates
> us_central.refresh.topics.interval.seconds=15
> us_central.refresh.groups.interval.seconds=15
> eu_west.refresh.topics.interval.seconds=15
> eu_west.refresh.groups.interval.seconds=15
> ## Sync options
> sync.topic.acls.enabled=true
> offset-syncs.topic.replication.enabled=true
> sync.group.offsets.enabled=true
> # EU pulls us, syd, nyc
> us_central->eu_west.enabled=true
> us_central->eu_west.sync.group.offsets.enabled=true
> # US pulls eu, syd, nyc
> eu_west->us_central.enabled=true
> eu_west->us_central.sync.group.offsets.enabled=false
> ```
> When starting MM2 freshly installed (deleted all mm2 related topics), we
> still get the excluded topics replicated between clusters.
> It's also worth mentioning that the connector seems to be reading the
> configuration correctly however it is (probably) at a later stage where the
> topics get filtered incorrectly.
> I attach a log from our connect.log where the exclude field is correctly
> parsed from our connect-mirrormaker.properties.
> ```
> groups.exclude = [console-consumer-.*, connect-.*, __.*]
> topics.exclude = [mm2_test_0]
> exclude.internal.topics = true
> ```
> After much testing and over 20 combinations of regex we have not found a way
> to exclude the topics via the configuration properties specified in the
> documentation for MM2.
> The *whitelist* does work though, so we are a bit confused in why the
> blacklist does not.
> We request help to further test this, we are not sure if we are doing
> something wrong.
>
> Thanks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
