[ https://issues.apache.org/jira/browse/KAFKA-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516755#comment-17516755 ]
David Bros edited comment on KAFKA-13796 at 4/4/22 11:49 AM: ------------------------------------------------------------- We solved the issue for now by not matching in the whitelist property, but this is not a very good solution: topics = ^(?!.\*(mm2_test_0|mm2_test_1)).\* was (Author: JIRAUSER284538): We solved the issue for now by not matching in the whitelist property, but this is not a very good solution: ^(?!.\*(mm2_test_0|mm2_test_1)).\* > MM2 - Topics Exclude/Blacklist not working > ------------------------------------------ > > Key: KAFKA-13796 > URL: https://issues.apache.org/jira/browse/KAFKA-13796 > Project: Kafka > Issue Type: Bug > Components: mirrormaker > Affects Versions: 3.0.0 > Reporter: David Bros > Priority: Blocker > Fix For: 3.0.0 > > > Hi all, > We are testing Kafka 3.0.0 in a development environment with Java 17 in a > Centos 7 machine. > When setting up MM2 replication between two clusters we exclude some topics > we don't want MM2 to replicate via topics.exclude (we have also tried > topics.blacklist) > Even though the configuration is parsed correctly, we still get those topics > replicated among clusters. > Here is our config for the us_central cluster: > ``` > # Clusters and bootstrap servers > clusters=eu_west, us_central > eu_west.bootstrap.servers=XXnet:9092,XX:9092 > us_central.bootstrap.servers=XX:9092 > us_central.consumer.auto.offset.reset=latest > us_central.consumer.fetch.max.bytes=31457280 > us_central.consumer.fetch.max.wait.ms=10000 > us_central.consumer.max.poll.records=3000 > us_central.consumer.request.timeout.ms=60000 > eu_west.consumer.auto.offset.reset=latest > eu_west.consumer.fetch.max.bytes=52428800 > eu_west.consumer.fetch.max.wait.ms=10000 > eu_west.consumer.max.poll.records=3000 > eu_west.consumer.request.timeout.ms=60000 > # Custom producer settings > us_central.producer.max.request.size=27262976 > us_central.producer.batch.size=22000 > us_central.producer.compression.type=none > us_central.producer.send.buffer.bytes=26214400 > us_central.producer.receive.buffer.bytes=26214400 > eu_west.producer.max.request.size=27262976 > eu_west.producer.batch.size=22000 > eu_west.producer.compression.type=none > eu_west.producer.send.buffer.bytes=26214400 > eu_west.producer.receive.buffer.bytes=26214400 > # Topics configuration > topics=.* > topics.blacklist="mm2_test_0" > # Tasks (threads) > tasks.max=10 > # Groups > groups=phx_netflow > # Replication factors, these are 1 for lab > replication.factor=1 > config.storage.replication.factor=1 > offset.storage.replication.factor=1 > status.storage.replication.factor=1 > checkpoints.topic.replication.factor=1 > offset-syncs.topic.replication.factor=1 > sync.group.offsets.replication.factor=1 > ## Config > config.properties.exclude=local.retention.ms > # Refresh rates > us_central.refresh.topics.interval.seconds=15 > us_central.refresh.groups.interval.seconds=15 > eu_west.refresh.topics.interval.seconds=15 > eu_west.refresh.groups.interval.seconds=15 > ## Sync options > sync.topic.acls.enabled=true > offset-syncs.topic.replication.enabled=true > sync.group.offsets.enabled=true > # EU pulls us, syd, nyc > us_central->eu_west.enabled=true > us_central->eu_west.sync.group.offsets.enabled=false > # US pulls eu, syd, nyc > eu_west->us_central.enabled=true > eu_west->us_central.sync.group.offsets.enabled=true > ``` > Here is the configuration for eu_west cluster: > ``` > # Clusters and bootstrap servers > clusters=eu_west, us_central > eu_west.bootstrap.servers=XX,XX:9092 > us_central.bootstrap.servers=XX:9092 > us_central.consumer.auto.offset.reset=latest > us_central.consumer.fetch.max.bytes=31457280 > us_central.consumer.fetch.max.wait.ms=10000 > us_central.consumer.max.poll.records=3000 > us_central.consumer.request.timeout.ms=60000 > eu_west.consumer.auto.offset.reset=latest > eu_west.consumer.fetch.max.bytes=52428800 > eu_west.consumer.fetch.max.wait.ms=10000 > eu_west.consumer.max.poll.records=3000 > eu_west.consumer.request.timeout.ms=60000 > # Custom producer settings > us_central.producer.max.request.size=27262976 > us_central.producer.batch.size=22000 > us_central.producer.compression.type=none > us_central.producer.send.buffer.bytes=26214400 > us_central.producer.receive.buffer.bytes=26214400 > eu_west.producer.max.request.size=27262976 > eu_west.producer.batch.size=22000 > eu_west.producer.compression.type=none > eu_west.producer.send.buffer.bytes=26214400 > eu_west.producer.receive.buffer.bytes=26214400 > # Topics configuration > topics=.* > topics.blacklist="mm2_test_0" > # Tasks (threads) > tasks.max=10 > # Groups > groups=ams_netflow > # Replication factors, these are 1 for lab > replication.factor=1 > config.storage.replication.factor=1 > offset.storage.replication.factor=1 > status.storage.replication.factor=1 > checkpoints.topic.replication.factor=1 > offset-syncs.topic.replication.factor=1 > sync.group.offsets.replication.factor=1 > ## Config > config.properties.exclude=local.retention.ms > # Refresh rates > us_central.refresh.topics.interval.seconds=15 > us_central.refresh.groups.interval.seconds=15 > eu_west.refresh.topics.interval.seconds=15 > eu_west.refresh.groups.interval.seconds=15 > ## Sync options > sync.topic.acls.enabled=true > offset-syncs.topic.replication.enabled=true > sync.group.offsets.enabled=true > # EU pulls us, syd, nyc > us_central->eu_west.enabled=true > us_central->eu_west.sync.group.offsets.enabled=true > # US pulls eu, syd, nyc > eu_west->us_central.enabled=true > eu_west->us_central.sync.group.offsets.enabled=false > ``` > When starting MM2 freshly installed (deleted all mm2 related topics), we > still get the excluded topics replicated between clusters. > It's also worth mentioning that the connector seems to be reading the > configuration correctly however it is (probably) at a later stage where the > topics get filtered incorrectly. > I attach a log from our connect.log where the exclude field is correctly > parsed from our connect-mirrormaker.properties. > ``` > groups.exclude = [console-consumer-.*, connect-.*, __.*] > topics.exclude = [mm2_test_0] > exclude.internal.topics = true > ``` > After much testing and over 20 combinations of regex we have not found a way > to exclude the topics via the configuration properties specified in the > documentation for MM2. > The *whitelist* does work though, so we are a bit confused in why the > blacklist does not. > We request help to further test this, we are not sure if we are doing > something wrong. > > Thanks -- This message was sent by Atlassian Jira (v8.20.1#820001)