Thank you very much Chris!
-Original Message-
From: Chris Egerton
Sent: Thursday, March 3, 2022 5:30 PM
To: users@kafka.apache.org
Subject: Re: Is MirrorMaker 2 horizontally scalable?
Hi Dmitri,
There's at least one issue with MirrorMaker 2 that impacts horizontal
scalability and has not yet been addressed:
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-9981&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=EKpH5nmW6OljbRz8HSUlrSp8f6p%2FZBeRPV6GpEJlLcU%3D&reserved=0.
There is some work in progress to fix it (
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-710%253A%2BFull%2Bsupport%2Bfor%2Bdistributed%2Bmode%2Bin%2Bdedicated%2BMirrorMaker%2B2.0%2Bclusters&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=6jTfhFhh3shJaS%2BPTTs5NJhFZ8uEXQ%2B1vlQfePecE08%3D&reserved=0),
but the effort hasn't received much attention to date.
There may be other issues as well, but until KAFKA-9981 is resolved, running
MirrorMaker 2 in a multi-node cluster will be at best difficult and at worst,
impossible.
Cheers,
Chris
On Thu, Mar 3, 2022 at 10:00 AM Dmitri Pavlov wrote:
> Hi,
>
> A quick question, maybe you can help?
>
> Trying to follow this article
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> i.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-382%253A%2BMirrorMak
> er%2B2.0&data=04%7C01%7Cdpavlov%40perforce.com%7C53b6338153194b386
> f5408d9fd2abb2b%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637819183
> 008636777%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi
> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FrA3yOCYijD6gP7lUDiD
> 6SutEg3cN%2Bo3Dywlr%2BIZsdk%3D&reserved=0
> -> "Walkthrough: Running MirrorMaker 2.0", and the last lines in the
> paragraph are
>
> ==
> Second, launch one or more MirrorMaker cluster nodes:
> $ ./bin/connect-mirror-maker.sh mm2.properties ==
>
> But apparently it does not work this way, one of 2 simultaneously
> started instance will remain idle, confirmed with Jconsole -> Mbeans.
> Setup: Broker A (in cluster A) -> MM2 2 instances -> Broker B (cluster
> B), for simplicity there is only one broker per cluster.
> A simple experiment is, when one instance is configured and started to
> replicate topic A only and another topic B only, only one topic will
> be replicated, when 2 instances are running in parallel. While, if
> only one of the instances is running at a time, each topic will be replicated
> correctly.
>
> The main question -> is Mirrormaker 2 horizontally scalable? And if
> yes, would be possible to share a link to a document that describes
> the setup process?
>
> Thanks in advance,
> Dmitri.
>
>
> This e-mail may contain information that is privileged or
> confidential. If you are not the intended recipient, please delete the
> e-mail and any attachments and notify us immediately.
>
>
CAUTION: This email originated from outside of the organization. Do not click
on links or open attachments unless you recognize the sender and know the
content is safe.
This e-mail may contain information that is privileged or confidential. If you
are not the intended recipient, please delete the e-mail and any attachments
and notify us immediately.
