Sebastian, there are multiple ways to run MM2. One way is to start the individual Connectors (MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector) on an existing Connect cluster, if you have one. Some of the configuration properties you've listed, e.g. "name" and "connector.class" are only relevant when configuring individual Connectors in this way.
The ./bin/connect-mirror.sh script, on the other hand, has its own high-level configuration file. This is where properties like "a->b.topics" come into play. The high-level configuration file is used to generate the low-level configurations for each internal Connector. Generally you'd use the high-level approach, unless you already have a bunch of Connect clusters you want to leverage. Keep this distinction in mind when looking at example configurations, and hopefully things will be clearer. Ryanne On Mon, Dec 23, 2019 at 1:34 PM Sebastian Schmitz < sebastian.schm...@propellerhead.co.nz> wrote: > Hello, > > I tried running this connect-mirror-config: > > <snip> > name = $MIRROR_NAME > clusters = source, target > source.bootstrap.servers = $SOURCE_SERVERS > target.bootstrap.servers = $TARGET_SERVERS > source->target.topics = $SOURCE_TARGET_TOPICS > target->source.topics = $TARGET_SOURCE_TOPICS > source->target.emit.heartbeats.enabled = true > target->source.emit.heartbeats.enabled = true > connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector > > # disable some new features > refresh.topics.enabled = false > refresh.groups.enabled = false > emit.checkpoints.enables = true > emit.heartbeats.enabled = true > sync.topic.configs.enabled = false > sync.topic.acls.enabled = false > </snip> > > SOURCE_SERVERS and TARGET_SERVERS are a comma-separated list of three > brokers with ports. > The TOPICS are |-separated lists of topics. > > I get these warning during startup which is a bit weird as I never > supplied any of those settings, but maybe I should? > > [2019-12-23 00:36:25,918] WARN The configuration 'config.storage.topic' > was supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,918] WARN The configuration > 'producer.bootstrap.servers' was supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,918] WARN The configuration 'group.id' was supplied > but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,919] WARN The configuration 'status.storage.topic' > was supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,919] WARN The configuration 'header.converter' was > supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,919] WARN The configuration > 'consumer.bootstrap.servers' was supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,919] WARN The configuration 'offset.storage.topic' > was supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,919] WARN The configuration 'value.converter' was > supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,919] WARN The configuration 'key.converter' was > supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > [2019-12-23 00:36:25,919] WARN The configuration > 'admin.bootstrap.servers' was supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig:355) > > And this error: > > [2019-12-23 00:36:29,320] ERROR Plugin class loader for connector: > 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found. > Returning: > org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@5c316230 > (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:165) > > First I tried the config mentioned in the KIP for "MirrorMaker Clusters" > which didn't work and I found removing the "cluster." from the > bootstrap-servers made it work a bit more, at least it didn't complain > about not having any servers in the config. > So, I checked the "Running a dedicated MirrorMaker cluster"from the KIP, > which is basically more or less the same, but without the "cluster." for > the servers and it does at least start and it looks like all the three > MMs find each other, but no mirroring taking place. > > Running the legacy-config from the old MM is working fine though. I'll > try to do some more digging today, so if you need some of those very > verbose logs or something else just let me know. I am sure that I can > figure this out and just wanted to know if the documentation will get > extended as the new MM2 has a lot of features and is a bit more > complicated than the old one... > > Thanks > > Sebastian > > On 24-Dec-19 8:06 AM, Ryanne Dolan wrote: > > Hello Sebastian, please let us know what issues you are facing and we can > > probably help. Which config from the KIP are you referencing? Also check > > out the readme under ./connect/mirror for more examples. > > > > Ryanne > > > > On Mon, Dec 23, 2019, 12:58 PM Sebastian Schmitz < > > sebastian.schm...@propellerhead.co.nz> wrote: > > > >> Hello, > >> > >> I'm currently trying to implement the new Kafka 2.4.0 and the new MM2. > >> > >> However, it looks like the only documentation available is the KIP-382, > >> and the documentation > >> (https://kafka.apache.org/documentation/#basic_ops_mirror_maker) for > the > >> MM isn't yet updated, and the documentation in the KIP seems to be > >> missing some stuff as I get a lot of errors and warning when starting > >> the MM2 as connect-mirror, and it doesn't mirror, so I probably have > >> some mistakes in my configuration, but can't confirm this as it's the > >> same as in the KIP. > >> > >> Any plans when the documentation will be updated? > >> > >> Thanks > >> > >> Sebastian > >> > >> > >> -- > >> DISCLAIMER > >> This email contains information that is confidential and which > >> may be > >> legally privileged. If you have received this email in error please > >> > >> notify the sender immediately and delete the email. > >> This email is intended > >> solely for the use of the intended recipient and you may not use or > >> disclose this email in any way. > >> > > -- > DISCLAIMER > This email contains information that is confidential and which > may be > legally privileged. If you have received this email in error please > > notify the sender immediately and delete the email. > This email is intended > solely for the use of the intended recipient and you may not use or > disclose this email in any way. >
