Running multiple instances of MM2
Hello! I’m wondering if it’s possible to scale up MM2 by running multiple instances per data center? For scalability purposes, I would like to run 2 instances of connect-mirror-maker.sh in one data center. However, I cannot get 2 instances of mirror maker to work at the same time: once I start up the second mirror maker, it takes over for the first one, and the first one completely stops replicating: i.e. rather than scaling up and rebalancing, the whole workload is still handled by a single instance. What am I missing here? Would greatly appreciate any guidance with this! Thank you! This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy. For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations. © 2022 BlackRock, Inc. All rights reserved.
Running multiple MM2 instances
Hello! I’m wondering if it’s possible to scale up MM2 by running multiple instances per data center? For scalability purposes, I would like to run 2 instances of connect-mirror-maker.sh in one data center. However, I cannot get 2 instances of mirror maker to work at the same time: once I start up the second mirror maker, it takes over for the first one, and the first one completely stops replicating: i.e. rather than scaling up and rebalancing, the whole workload is still handled by a single instance. What am I missing here? Would greatly appreciate any guidance with this! Thank you! This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy. For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations. © 2022 BlackRock, Inc. All rights reserved.
Re: Running multiple MM2 instances
Hi Julia, Sounds like KAFKA-9981 [1]. This is a known issue with MirrorMaker 2 that impacts horizontal scalability and has not yet been addressed. There is some work in progress to fix this issue [2], but the effort hasn't received much attention to date. There may be other issues as well, but until KAFKA-9981 is addressed, running MirrorMaker 2 in a multi-node cluster will be at best difficult and worst, impossible. [1] - https://issues.apache.org/jira/browse/KAFKA-9981 [2] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters Cheers, Chris On Wed, Mar 23, 2022 at 11:15 AM Kalimova, Julia wrote: > > > Hello! > > > > I’m wondering if it’s possible to scale up MM2 by running multiple > instances per data center? > > For scalability purposes, I would like to run 2 instances of > connect-mirror-maker.sh in one data center. However, I cannot get 2 > instances of mirror maker to work at the same time: once I start up the > second mirror maker, it takes over for the first one, and the first one > completely stops replicating: i.e. rather than scaling up and rebalancing, > the whole workload is still handled by a single instance. > > What am I missing here? Would greatly appreciate any guidance with this! > Thank you! > > > This message may contain information that is confidential or privileged. > If you are not the intended recipient, please advise the sender immediately > and delete this message. See > http://www.blackrock.com/corporate/compliance/email-disclaimers for > further information. Please refer to > http://www.blackrock.com/corporate/compliance/privacy-policy for more > information about BlackRock’s Privacy Policy. > > > For a list of BlackRock's office addresses worldwide, see > http://www.blackrock.com/corporate/about-us/contacts-locations. > > © 2022 BlackRock, Inc. All rights reserved. >
