[
https://issues.apache.org/jira/browse/KAFKA-17851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892681#comment-17892681
]
Zhijian Chen commented on KAFKA-17851:
--------------------------------------
[~gharris1727] Thanks for reply.
There is a scenario where cluster A topic1 is replicated to cluster B's
A.topic1, and then cluster B's A.topic1 has consumerB that has been consuming.
If cluster B hangs, A.topic1 is affected and all replication meta-information
is lost, so no one knows where consumerB has been consuming.
If I have a separate cluster that keeps the replication meta-information, then
even if cluster B hangs, the replication meta-information is still there, and I
can query and rebuild a replication flow based on this information, A to C's
A.topic1, and consumerB consumes the A.topic1 of C, and starts consuming from
the correct location.
This independent cluster can be deployed using multiple server rooms to
increase its reliability.
Simply put, a standalone cluster allows for a bit more availability of
replication meta-information.
> Save internal topic to a separate kafka cluster
> -----------------------------------------------
>
> Key: KAFKA-17851
> URL: https://issues.apache.org/jira/browse/KAFKA-17851
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Zhijian Chen
> Priority: Major
>
> For the mm2-offset-syncs topic there is a parameter to control its storage
> cluster: offset-syncs.topic.location, but this parameter can only control
> whether it is in the source or target cluster.
> For checkpoints topic and heartbeat topic, there is no parameter to control
> their storage clusters.
> In a multi-cluster backup scenario, we would like to store these internal
> topics in a separate kafka cluster, which has the advantage of being easy to
> manage, and we can deploy this separate cluster across three server rooms so
> that we don't have to worry about the failure of this separate cluster, which
> also solves the problem of the existing scenario, where the internal topics
> cannot be accessed due to the failure of the source or target clusters.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
