ning2008wisc commented on a change in pull request #9215: URL: https://github.com/apache/kafka/pull/9215#discussion_r478583076
########## File path: connect/mirror/README.md ########## @@ -141,7 +141,38 @@ nearby clusters. N.B. that the `--clusters` parameter is not technically required here. MM2 will work fine without it; however, throughput may suffer from "producer lag" between data centers, and you may incur unnecessary data transfer costs. -## Shared configuration +## Configuration +### General Kafka Connect Config +All Kafka Connect, Source Connector, Sink Connector configs, as defined in [Kafka official doc] (https://kafka.apache.org/documentation/#connectconfigs), can be Review comment: done ########## File path: connect/mirror/README.md ########## @@ -141,7 +141,38 @@ nearby clusters. N.B. that the `--clusters` parameter is not technically required here. MM2 will work fine without it; however, throughput may suffer from "producer lag" between data centers, and you may incur unnecessary data transfer costs. -## Shared configuration +## Configuration +### General Kafka Connect Config +All Kafka Connect, Source Connector, Sink Connector configs, as defined in [Kafka official doc] (https://kafka.apache.org/documentation/#connectconfigs), can be +directly used in MM2 configuration without prefix in the configuration name. As the starting point, most of these default configs may work well with the exception of `tasks.max`. + +In order to evenly distribute the workload across more than one MM2 instance, it is advised to set `tasks.max` at least to 2 or even larger depending on the hardware resources +and the total number partitions to be replicated. + +### Kafka Connect Config for a Specific Connector +If needed, Kafka Connect worker-level configs could be even specified "per connector", which needs to follow the format of `cluster_alias.config_name` in MM2 configuration. For example, + + backup.ssl.truststore.location = /usr/lib/jvm/zulu-8-amd64/jre/lib/security/cacerts // SSL cert location + backup.security.protocol = SSL // if target cluster needs SSL to send message + +### MM2 configs for a Specific Connector +MM2 itself has many configs to control how MM2 behaves. To override those default values, add the config name by the format of `source_cluster_alias->target_cluster_alias.config_name` in MM2 configuration. For example, Review comment: done ########## File path: connect/mirror/README.md ########## @@ -141,7 +141,38 @@ nearby clusters. N.B. that the `--clusters` parameter is not technically required here. MM2 will work fine without it; however, throughput may suffer from "producer lag" between data centers, and you may incur unnecessary data transfer costs. -## Shared configuration +## Configuration +### General Kafka Connect Config +All Kafka Connect, Source Connector, Sink Connector configs, as defined in [Kafka official doc] (https://kafka.apache.org/documentation/#connectconfigs), can be +directly used in MM2 configuration without prefix in the configuration name. As the starting point, most of these default configs may work well with the exception of `tasks.max`. + +In order to evenly distribute the workload across more than one MM2 instance, it is advised to set `tasks.max` at least to 2 or even larger depending on the hardware resources +and the total number partitions to be replicated. + +### Kafka Connect Config for a Specific Connector +If needed, Kafka Connect worker-level configs could be even specified "per connector", which needs to follow the format of `cluster_alias.config_name` in MM2 configuration. For example, + + backup.ssl.truststore.location = /usr/lib/jvm/zulu-8-amd64/jre/lib/security/cacerts // SSL cert location + backup.security.protocol = SSL // if target cluster needs SSL to send message + +### MM2 configs for a Specific Connector +MM2 itself has many configs to control how MM2 behaves. To override those default values, add the config name by the format of `source_cluster_alias->target_cluster_alias.config_name` in MM2 configuration. For example, + + backup->primary.enabled = false // set to false if one-way replication is desired + primary->backup.topics.blacklist = topics_to_blacklist + primary->backup.emit.heartbeats.enabled = false + primary->backup.sync.group.offsets = true + +### Producer / Consumer Config used by MM2 Review comment: covered Admin in the latest commit ########## File path: connect/mirror/README.md ########## @@ -141,7 +141,38 @@ nearby clusters. N.B. that the `--clusters` parameter is not technically required here. MM2 will work fine without it; however, throughput may suffer from "producer lag" between data centers, and you may incur unnecessary data transfer costs. -## Shared configuration +## Configuration Review comment: "driver mode" means dedicated MM2 cluster? If yes, any difference on configuration name (e.g. prefix) between running dedicated MM2 cluster or MM2 on existing KC cluster? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
