David Bros created KAFKA-13796:
----------------------------------
Summary: MM2 - Topics Exclude/Blacklist not working
Key: KAFKA-13796
URL: https://issues.apache.org/jira/browse/KAFKA-13796
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 3.0.0
Reporter: David Bros
Fix For: 3.0.0
Hi all,
We are testing Kafka 3.0.0 in a development environment with Java 17 in a
Centos 7 machine.
When setting up MM2 replication between two clusters we exclude some topics we
don't want MM2 to replicate via topics.exclude (we have also tried
topics.blacklist)
Even though the configuration is parsed correctly, we still get those topics
replicated among clusters.
Here is our config for the us_central cluster:
```
# Clusters and bootstrap servers
clusters=eu_west, us_central
eu_west.bootstrap.servers=XXnet:9092,XX:9092
us_central.bootstrap.servers=XX:9092
us_central.consumer.auto.offset.reset=latest
us_central.consumer.fetch.max.bytes=31457280
us_central.consumer.fetch.max.wait.ms=10000
us_central.consumer.max.poll.records=3000
us_central.consumer.request.timeout.ms=60000
eu_west.consumer.auto.offset.reset=latest
eu_west.consumer.fetch.max.bytes=52428800
eu_west.consumer.fetch.max.wait.ms=10000
eu_west.consumer.max.poll.records=3000
eu_west.consumer.request.timeout.ms=60000
# Custom producer settings
us_central.producer.max.request.size=27262976
us_central.producer.batch.size=22000
us_central.producer.compression.type=none
us_central.producer.send.buffer.bytes=26214400
us_central.producer.receive.buffer.bytes=26214400
eu_west.producer.max.request.size=27262976
eu_west.producer.batch.size=22000
eu_west.producer.compression.type=none
eu_west.producer.send.buffer.bytes=26214400
eu_west.producer.receive.buffer.bytes=26214400
# Topics configuration
topics=.*
topics.blacklist="mm2_test_0"
# Tasks (threads)
tasks.max=10
# Groups
groups=phx_netflow
# Replication factors, these are 1 for lab
replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
checkpoints.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
sync.group.offsets.replication.factor=1
## Config
config.properties.exclude=local.retention.ms
# Refresh rates
us_central.refresh.topics.interval.seconds=15
us_central.refresh.groups.interval.seconds=15
eu_west.refresh.topics.interval.seconds=15
eu_west.refresh.groups.interval.seconds=15
## Sync options
sync.topic.acls.enabled=true
offset-syncs.topic.replication.enabled=true
sync.group.offsets.enabled=true
# EU pulls us, syd, nyc
us_central->eu_west.enabled=true
us_central->eu_west.sync.group.offsets.enabled=false
# US pulls eu, syd, nyc
eu_west->us_central.enabled=true
eu_west->us_central.sync.group.offsets.enabled=true
```
Here is the configuration for eu_west cluster:
```
# Clusters and bootstrap servers
clusters=eu_west, us_central
eu_west.bootstrap.servers=XX,XX:9092
us_central.bootstrap.servers=XX:9092
us_central.consumer.auto.offset.reset=latest
us_central.consumer.fetch.max.bytes=31457280
us_central.consumer.fetch.max.wait.ms=10000
us_central.consumer.max.poll.records=3000
us_central.consumer.request.timeout.ms=60000
eu_west.consumer.auto.offset.reset=latest
eu_west.consumer.fetch.max.bytes=52428800
eu_west.consumer.fetch.max.wait.ms=10000
eu_west.consumer.max.poll.records=3000
eu_west.consumer.request.timeout.ms=60000
# Custom producer settings
us_central.producer.max.request.size=27262976
us_central.producer.batch.size=22000
us_central.producer.compression.type=none
us_central.producer.send.buffer.bytes=26214400
us_central.producer.receive.buffer.bytes=26214400
eu_west.producer.max.request.size=27262976
eu_west.producer.batch.size=22000
eu_west.producer.compression.type=none
eu_west.producer.send.buffer.bytes=26214400
eu_west.producer.receive.buffer.bytes=26214400
# Topics configuration
topics=.*
topics.blacklist="mm2_test_0"
# Tasks (threads)
tasks.max=10
# Groups
groups=ams_netflow
# Replication factors, these are 1 for lab
replication.factor=1
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
checkpoints.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
sync.group.offsets.replication.factor=1
## Config
config.properties.exclude=local.retention.ms
# Refresh rates
us_central.refresh.topics.interval.seconds=15
us_central.refresh.groups.interval.seconds=15
eu_west.refresh.topics.interval.seconds=15
eu_west.refresh.groups.interval.seconds=15
## Sync options
sync.topic.acls.enabled=true
offset-syncs.topic.replication.enabled=true
sync.group.offsets.enabled=true
# EU pulls us, syd, nyc
us_central->eu_west.enabled=true
us_central->eu_west.sync.group.offsets.enabled=true
# US pulls eu, syd, nyc
eu_west->us_central.enabled=true
eu_west->us_central.sync.group.offsets.enabled=false
```
When starting MM2 freshly installed (deleted all mm2 related topics), we still
get the excluded topics replicated between clusters.
It's also worth mentioning that the connector seems to be reading the
configuration correctly however it is (probably) at a later stage where the
topics get filtered incorrectly.
I attach a log from our connect.log where the exclude field is correctly parsed
from our connect-mirrormaker.properties.
```
groups.exclude = [console-consumer-.*, connect-.*, __.*]
topics.exclude = [mm2_test_0]
exclude.internal.topics = true
```
After much testing and over 20 combinations of regex we have not found a way to
exclude the topics via the configuration properties specified in the
documentation for MM2.
The *whitelist* does work though, so we are a bit confused in why the blacklist
does not.
We request help to further test this, we are not sure if we are doing something
wrong.
Thanks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
