[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275292#comment-17275292 ] A. Sophie Blee-Goldman commented on KAFKA-8930: --- Hey [~miguno] I changed the FixVersion to 2.6.2 since it looks like this was just merged a few days ago. Let me know if that isn't right > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Michael G. Noll >Priority: Minor > Fix For: 2.7.0, 2.6.2 > > > *New description (completed):* > Add user-facing documentation (for kafka.apache.org) on how to perform > geo-replication and cross-cluster data replication/mirroring with the new > MirrorMaker v2, along with example usage and example configuration. > The new documentation is available at > https://kafka.apache.org/documentation/#georeplication. > *Old description (not done):* > * Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > * Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273075#comment-17273075 ] ASF GitHub Bot commented on KAFKA-8930: --- bbejeck merged pull request #326: URL: https://github.com/apache/kafka-site/pull/326 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Michael G. Noll >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273074#comment-17273074 ] ASF GitHub Bot commented on KAFKA-8930: --- bbejeck edited a comment on pull request #326: URL: https://github.com/apache/kafka-site/pull/326#issuecomment-768481852 > I didn't test this PR locally myself to ensure proper HTML rendering etc. FWIW I rendered it locally and it seemed fine This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Michael G. Noll >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273073#comment-17273073 ] ASF GitHub Bot commented on KAFKA-8930: --- bbejeck commented on pull request #326: URL: https://github.com/apache/kafka-site/pull/326#issuecomment-768481852 > I didn't test this PR locally myself to ensure proper HTML rendering etc. FWIW I rendered it locally and it seemed fine This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Michael G. Noll >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273059#comment-17273059 ] ASF GitHub Bot commented on KAFKA-8930: --- miguno commented on pull request #326: URL: https://github.com/apache/kafka-site/pull/326#issuecomment-768466468 This LGTM, though (1) there were some minor HTML changes not directly related to the original PR and (2) I didn't test this PR locally myself to ensure proper HTML rendering etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Michael G. Noll >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273055#comment-17273055 ] ASF GitHub Bot commented on KAFKA-8930: --- bbejeck commented on pull request #326: URL: https://github.com/apache/kafka-site/pull/326#issuecomment-768461055 ping @miguno for a +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Michael G. Noll >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273030#comment-17273030 ] ASF GitHub Bot commented on KAFKA-8930: --- bbejeck opened a new pull request #326: URL: https://github.com/apache/kafka-site/pull/326 The MM2 docs are already in for 2.7 via https://github.com/apache/kafka-site/pull/324, this PR adds them to 2.6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Michael G. Noll >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272987#comment-17272987 ] ASF GitHub Bot commented on KAFKA-8930: --- miguno commented on pull request #324: URL: https://github.com/apache/kafka-site/pull/324#issuecomment-768416513 kafka/docs PR is up at https://github.com/apache/kafka/pull/9983 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272977#comment-17272977 ] ASF GitHub Bot commented on KAFKA-8930: --- bbejeck commented on pull request #324: URL: https://github.com/apache/kafka-site/pull/324#issuecomment-768404929 > We should also add these docs to kafka/docs repo @omkreddy, yes a PR for kafka/docs is coming soon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272942#comment-17272942 ] ASF GitHub Bot commented on KAFKA-8930: --- omkreddy commented on pull request #324: URL: https://github.com/apache/kafka-site/pull/324#issuecomment-768372236 We should also add these docs to `kafka/docs` repo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272938#comment-17272938 ] ASF GitHub Bot commented on KAFKA-8930: --- bbejeck merged pull request #324: URL: https://github.com/apache/kafka-site/pull/324 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[
https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271863#comment-17271863
]
ASF GitHub Bot commented on KAFKA-8930:
---
miguno commented on a change in pull request #324:
URL: https://github.com/apache/kafka-site/pull/324#discussion_r563562309
##
File path: 27/ops.html
##
@@ -553,7 +539,558 @@ 6.3 Kafka Configuration
+ 6.3 Geo-Replication (Cross-Cluster Data
Mirroring)
+
+ Geo-Replication
Overview
+
+
+Kafka administrators can define data flows that cross the boundaries of
individual Kafka clusters, data centers, or geo-regions. Such event streaming
setups are often needed for organizational, technical, or legal requirements.
Common scenarios include:
+
+
+
+Geo-replication
+Disaster recovery
+Feeding edge clusters into a central, aggregate cluster
+Physical isolation of clusters (such as production vs. testing)
+Cloud migration or hybrid cloud deployments
+Legal and compliance requirements
+
+
+
+Administrators can set up such inter-cluster data flows with Kafka's
MirrorMaker (version 2), a tool to replicate data between different Kafka
environments in a streaming manner. MirrorMaker is built on top of the Kafka
Connect framework and supports features such as:
+
+
+
+Replicates topics (data plus configurations)
+Replicates consumer groups including offsets to migrate applications
between clusters
+Replicates ACLs
+Preserves partitioning
+Automatically detects new topics and partitions
+Provides a wide range of metrics, such as end-to-end replication
latency across multiple data centers/clusters
+Fault-tolerant and horizontally scalable operations
+
+
+
+ Note: Geo-replication with MirrorMaker replicates data across Kafka
clusters. This inter-cluster replication is different from Kafka's intra-cluster replication, which replicates data within
the same Kafka cluster.
+
+
+ What Are Replication
Flows
+
+
+With MirrorMaker, Kafka administrators can replicate topics, topic
configurations, consumer groups and their offsets, and ACLs from one or more
source Kafka clusters to one or more target Kafka clusters, i.e., across
cluster environments. In a nutshell, MirrorMaker consumes data from the source
cluster with source connectors, and then replicates the data by producing to
the target cluster with sink connectors.
+
+
+
+These directional flows from source to target clusters are called
replication flows. They are defined with the format
{source_cluster}->{target_cluster} in the MirrorMaker
configuration file as described later. Administrators can create complex
replication topologies based on these flows.
+
+
+
+Here are some example patterns:
+
+
+
+Active/Active high availability deployments: A->B,
B->A
+Active/Passive or Active/Standby high availability deployments:
A->B
+Aggregation (e.g., from many clusters to one): A->K, B->K,
C->K
+Fan-out (e.g., from one to many clusters): K->A, K->B,
K->C
+Forwarding: A->B, B->C, C->D
+
+
+
+By default, a flow replicates all topics and consumer groups. However,
each replication flow can be configured independently. For instance, you can
define that only specific topics or consumer groups are replicated from the
source cluster to the target cluster.
+
+
+
+Here is a first example on how to configure data replication from a
primary cluster to a secondary cluster (an
active/passive setup):
+
+
+# Basic settings
+clusters = primary, secondary
+primary.bootstrap.servers = broker3-primary:9092
+secondary.bootstrap.servers = broker5-secondary:9092
+
+# Define replication flows
+primary->secondary.enable = true
+primary->secondary.topics = foobar-topic, quux-.*
+
+
+
+ Configuring
Geo-Replication
+
+
+The following sections describe how to configure and run a dedicated
MirrorMaker cluster. If you want to run MirrorMaker within an existing Kafka
Connect cluster or other supported deployment setups, please refer to https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0";>KIP-382:
MirrorMaker 2.0 and be aware that the names of configuration settings may
vary between deployment modes.
+
+
+
+Beyond what's covered in the following sections, further examples and
information on configuration settings are available at:
+
+
+
+ https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMakerConfig.java";>MirrorMakerConfig,
https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java";>MirrorConnectorConfig
+ https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/DefaultTopicFilter.java";>DefaultTopicFilter
for topics, https://
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271182#comment-17271182 ] ASF GitHub Bot commented on KAFKA-8930: --- miguno commented on a change in pull request #324: URL: https://github.com/apache/kafka-site/pull/324#discussion_r563562851 ## File path: 27/ops.html ## @@ -553,7 +539,558 @@ 6.3 Kafka Configuration + 6.3 Geo-Replication (Cross-Cluster Data Mirroring) + + Geo-Replication Overview + + +Kafka administrators can define data flows that cross the boundaries of individual Kafka clusters, data centers, or geo-regions. Such event streaming setups are often needed for organizational, technical, or legal requirements. Common scenarios include: + + + +Geo-replication +Disaster recovery +Feeding edge clusters into a central, aggregate cluster +Physical isolation of clusters (such as production vs. testing) +Cloud migration or hybrid cloud deployments +Legal and compliance requirements + + + +Administrators can set up such inter-cluster data flows with Kafka's MirrorMaker (version 2), a tool to replicate data between different Kafka environments in a streaming manner. MirrorMaker is built on top of the Kafka Connect framework and supports features such as: + + + +Replicates topics (data plus configurations) +Replicates consumer groups including offsets to migrate applications between clusters +Replicates ACLs +Preserves partitioning +Automatically detects new topics and partitions +Provides a wide range of metrics, such as end-to-end replication latency across multiple data centers/clusters +Fault-tolerant and horizontally scalable operations + + + + Note: Geo-replication with MirrorMaker replicates data across Kafka clusters. This inter-cluster replication is different from Kafka's intra-cluster replication, which replicates data within the same Kafka cluster. + + + What Are Replication Flows + + +With MirrorMaker, Kafka administrators can replicate topics, topic configurations, consumer groups and their offsets, and ACLs from one or more source Kafka clusters to one or more target Kafka clusters, i.e., across cluster environments. In a nutshell, MirrorMaker consumes data from the source cluster with source connectors, and then replicates the data by producing to the target cluster with sink connectors. Review comment: Thanks, @ryannedolan. Text updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[
https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271180#comment-17271180
]
ASF GitHub Bot commented on KAFKA-8930:
---
miguno commented on a change in pull request #324:
URL: https://github.com/apache/kafka-site/pull/324#discussion_r563562309
##
File path: 27/ops.html
##
@@ -553,7 +539,558 @@ 6.3 Kafka Configuration
+ 6.3 Geo-Replication (Cross-Cluster Data
Mirroring)
+
+ Geo-Replication
Overview
+
+
+Kafka administrators can define data flows that cross the boundaries of
individual Kafka clusters, data centers, or geo-regions. Such event streaming
setups are often needed for organizational, technical, or legal requirements.
Common scenarios include:
+
+
+
+Geo-replication
+Disaster recovery
+Feeding edge clusters into a central, aggregate cluster
+Physical isolation of clusters (such as production vs. testing)
+Cloud migration or hybrid cloud deployments
+Legal and compliance requirements
+
+
+
+Administrators can set up such inter-cluster data flows with Kafka's
MirrorMaker (version 2), a tool to replicate data between different Kafka
environments in a streaming manner. MirrorMaker is built on top of the Kafka
Connect framework and supports features such as:
+
+
+
+Replicates topics (data plus configurations)
+Replicates consumer groups including offsets to migrate applications
between clusters
+Replicates ACLs
+Preserves partitioning
+Automatically detects new topics and partitions
+Provides a wide range of metrics, such as end-to-end replication
latency across multiple data centers/clusters
+Fault-tolerant and horizontally scalable operations
+
+
+
+ Note: Geo-replication with MirrorMaker replicates data across Kafka
clusters. This inter-cluster replication is different from Kafka's intra-cluster replication, which replicates data within
the same Kafka cluster.
+
+
+ What Are Replication
Flows
+
+
+With MirrorMaker, Kafka administrators can replicate topics, topic
configurations, consumer groups and their offsets, and ACLs from one or more
source Kafka clusters to one or more target Kafka clusters, i.e., across
cluster environments. In a nutshell, MirrorMaker consumes data from the source
cluster with source connectors, and then replicates the data by producing to
the target cluster with sink connectors.
+
+
+
+These directional flows from source to target clusters are called
replication flows. They are defined with the format
{source_cluster}->{target_cluster} in the MirrorMaker
configuration file as described later. Administrators can create complex
replication topologies based on these flows.
+
+
+
+Here are some example patterns:
+
+
+
+Active/Active high availability deployments: A->B,
B->A
+Active/Passive or Active/Standby high availability deployments:
A->B
+Aggregation (e.g., from many clusters to one): A->K, B->K,
C->K
+Fan-out (e.g., from one to many clusters): K->A, K->B,
K->C
+Forwarding: A->B, B->C, C->D
+
+
+
+By default, a flow replicates all topics and consumer groups. However,
each replication flow can be configured independently. For instance, you can
define that only specific topics or consumer groups are replicated from the
source cluster to the target cluster.
+
+
+
+Here is a first example on how to configure data replication from a
primary cluster to a secondary cluster (an
active/passive setup):
+
+
+# Basic settings
+clusters = primary, secondary
+primary.bootstrap.servers = broker3-primary:9092
+secondary.bootstrap.servers = broker5-secondary:9092
+
+# Define replication flows
+primary->secondary.enable = true
+primary->secondary.topics = foobar-topic, quux-.*
+
+
+
+ Configuring
Geo-Replication
+
+
+The following sections describe how to configure and run a dedicated
MirrorMaker cluster. If you want to run MirrorMaker within an existing Kafka
Connect cluster or other supported deployment setups, please refer to https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0";>KIP-382:
MirrorMaker 2.0 and be aware that the names of configuration settings may
vary between deployment modes.
+
+
+
+Beyond what's covered in the following sections, further examples and
information on configuration settings are available at:
+
+
+
+ https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMakerConfig.java";>MirrorMakerConfig,
https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java";>MirrorConnectorConfig
+ https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/DefaultTopicFilter.java";>DefaultTopicFilter
for topics, https://
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270731#comment-17270731 ] ASF GitHub Bot commented on KAFKA-8930: --- ryannedolan commented on a change in pull request #324: URL: https://github.com/apache/kafka-site/pull/324#discussion_r563180323 ## File path: 27/ops.html ## @@ -553,7 +539,558 @@ 6.3 Kafka Configuration + 6.3 Geo-Replication (Cross-Cluster Data Mirroring) + + Geo-Replication Overview + + +Kafka administrators can define data flows that cross the boundaries of individual Kafka clusters, data centers, or geo-regions. Such event streaming setups are often needed for organizational, technical, or legal requirements. Common scenarios include: + + + +Geo-replication +Disaster recovery +Feeding edge clusters into a central, aggregate cluster +Physical isolation of clusters (such as production vs. testing) +Cloud migration or hybrid cloud deployments +Legal and compliance requirements + + + +Administrators can set up such inter-cluster data flows with Kafka's MirrorMaker (version 2), a tool to replicate data between different Kafka environments in a streaming manner. MirrorMaker is built on top of the Kafka Connect framework and supports features such as: + + + +Replicates topics (data plus configurations) +Replicates consumer groups including offsets to migrate applications between clusters +Replicates ACLs +Preserves partitioning +Automatically detects new topics and partitions +Provides a wide range of metrics, such as end-to-end replication latency across multiple data centers/clusters +Fault-tolerant and horizontally scalable operations + + + + Note: Geo-replication with MirrorMaker replicates data across Kafka clusters. This inter-cluster replication is different from Kafka's intra-cluster replication, which replicates data within the same Kafka cluster. + + + What Are Replication Flows + + +With MirrorMaker, Kafka administrators can replicate topics, topic configurations, consumer groups and their offsets, and ACLs from one or more source Kafka clusters to one or more target Kafka clusters, i.e., across cluster environments. In a nutshell, MirrorMaker consumes data from the source cluster with source connectors, and then replicates the data by producing to the target cluster with sink connectors. Review comment: "with sink connectors" is not true at the moment, since I don't think we have a sink connector yet. And even when we do, it would usually be sufficient to use source _or_ sink connector. There are certainly cases where this sentence is true, but I think it's misleading as a general statement. Maybe "In a nutshell, MirrorMaker uses Connectors to consume from source clusters and produce to target clusters" or something like that. ## File path: 27/ops.html ## @@ -553,7 +539,558 @@ 6.3 Kafka Configuration + 6.3 Geo-Replication (Cross-Cluster Data Mirroring) + + Geo-Replication Overview + + +Kafka administrators can define data flows that cross the boundaries of individual Kafka clusters, data centers, or geo-regions. Such event streaming setups are often needed for organizational, technical, or legal requirements. Common scenarios include: + + + +Geo-replication +Disaster recovery +Feeding edge clusters into a central, aggregate cluster +Physical isolation of clusters (such as production vs. testing) +Cloud migration or hybrid cloud deployments +Legal and compliance requirements + + + +Administrators can set up such inter-cluster data flows with Kafka's MirrorMaker (version 2), a tool to replicate data between different Kafka environments in a streaming manner. MirrorMaker is built on top of the Kafka Connect framework and supports features such as: + + + +Replicates topics (data plus configurations) +Replicates consumer groups including offsets to migrate applications between clusters +Replicates ACLs +Preserves partitioning +Automatically detects new topics and partitions +Provides a wide range of metrics, such as end-to-end replication latency across multiple data centers/clusters +Fault-tolerant and horizontally scalable operations + + + + Note: Geo-replication with MirrorMaker replicates data across Kafka clusters. This inter-cluster replication is different from Kafka's intra-cluster replication, which replicates data within the same Kafka cluster. + + + What Are Replication Flows + + +With MirrorMaker, Kafka administrators can replicate topics, topic configurations, consumer groups and their offsets, and ACLs from one or more source Kafka clusters to one or more target Kafka clusters, i.e., across cluster environments. In a nutshell, MirrorMaker consumes data from the source clu
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270228#comment-17270228 ] Michael G. Noll commented on KAFKA-8930: I submitted a PR for the user-facing documentation (i.e., .not JavaDocs): https://github.com/apache/kafka-site/pull/324 > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270225#comment-17270225 ] ASF GitHub Bot commented on KAFKA-8930: --- miguno opened a new pull request #324: URL: https://github.com/apache/kafka-site/pull/324 This adds a new user-facing documentation "Geo-replication (Cross-Cluster Data Mirroring)" section to the Kafka Operations documentation that covers MirrorMaker v2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245833#comment-17245833 ] Michael G. Noll commented on KAFKA-8930: Why is the ticket status "Patch Available" when it seems that no work has started yet? > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17217765#comment-17217765 ] Bill Bejeck commented on KAFKA-8930: As part of the 2.7.0 release process, I'm clearing the fix version field. When we have a PR for this we can set the fix version at that point. > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Fix For: 2.7.0 > > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8930) MM2 documentation
[ https://issues.apache.org/jira/browse/KAFKA-8930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118200#comment-17118200 ] Rodrigo Alejandro Loza Lucero commented on KAFKA-8930: -- Sorry for the question, but I couldn't find the javadocs. Could anyone point me to them? Thanks. > MM2 documentation > - > > Key: KAFKA-8930 > URL: https://issues.apache.org/jira/browse/KAFKA-8930 > Project: Kafka > Issue Type: Improvement > Components: documentation, mirrormaker >Affects Versions: 2.4.0 >Reporter: Ryanne Dolan >Assignee: Ryanne Dolan >Priority: Minor > Fix For: 2.6.0 > > > Expand javadocs for new MirrorMaker (entrypoint) and MirrorMakerConfig > classes. Include example usage and example configuration. > Expand javadocs for MirrorSourceConnector, MirrorCheckpointConnector, and > MirrorHeartbeatConnector, including example configuration for running on > Connect w/o mm2 driver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
