So we've just hit this issue again just with the MM2 connector and trying to
add a new mirrored topic.We're running MirrorMaker 2 in Strimzi. i.e.
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector"We
have 6 worker nodes.We changed the config to add a new mirror topic. i.e.
append a new topic to the MirrorSourceConnector's "topics" config.The MM2
config topic reflects the change, as does viewing the config using Kowl
UI.However, no tasks run to mirror the newly-added topic.We also do not see any
updates on the MM2 status topic for the mirroring of that newly-added topic.
We tried adding tasks to trigger a propagation of the task configs (increased
from 36 to 40 tasks) however that did not unblock it.So triggering this code
path did not seem to work:
https://github.com/apache/kafka/blob/8cb0a5e9d3441962896b79163d141607e94d9b54/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1569-L1572
Only restarting the workers seemed to unblock the propagation of the new task
config for the new mirrored topic.
Hopefully this can help us narrow things down a bit...
In the meantime we've since enabled the following DEBUG logging in production
to try to get more hints the next time this happens:
log4j.logger.org.apache.kafka.connect.storage.KafkaConfigBackingStore: DEBUG
log4j.logger.org.apache.kafka.connect.runtime.distributed.DistributedHerder:
DEBUG
Perhaps that will show us if it's at all related MM2 config topic compaction
and/or connectors in inconsistent state.
Are there any other (not overly-verbose) classes you recommend we enable DEBUG
logging on?
Also, would making the inconsistent connectors (assuming they're being
identified as such by Kafka Connect when this happens) through an API call also
make sense so that this can be detected/monitored more easily?
Thanks!
