Thanks Ewen and Guozhang for the details ! On Thu, Dec 10, 2015 at 10:31 PM, Guozhang Wang <wangg...@gmail.com> wrote:
> Meghana, > > As Ewen said in KAFKA-1650 we removed the support of multiple consumer > configs in MM, so that you cannot get one MM instance to pull from multiple > origin clusters, but instead you need to use one MM instance for each > origin. > > The motivation for dropping this support is mainly operational: with the > previous settings if one of the original cluster falls into a bad state, we > cannot stop / debug that MM since it also pulls from other clusters. And > other minor issues include metics reporting, etc. For example at LI people > use one MM instance for each origin cluster, that are co-located with the > destination cluster, so if you need to mirror from cluster A / B / C to D, > you will have a A->D / B->D / C->D MM. Moving forward we are also trying to > add exact mirroring features into MM, which would better be a 1-to-1 > mapping. > > Guozhang > > > On Thu, Dec 10, 2015 at 3:21 PM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > Meghana, > > > > It looks like this functionality was removed in > > https://issues.apache.org/jira/browse/KAFKA-1650, although I don't see > > explicit discussion of the issue in the JIRA so I'm not sure of the exact > > motivation. Maybe Becket or Guozhang can offer some insight about if it > is > > necessary (that JIRA fixes a data loss issue) or an accidental > > incompatibility. > > > > -Ewen > > > > On Thu, Dec 10, 2015 at 2:32 PM, Meghana Narasimhan < > > mnarasim...@bandwidth.com> wrote: > > > > > Hi, > > > > > > We upgraded to the Confluent 2.0 platform today (we were earlier using > > the > > > Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 > that > > > is packaged with the platform we are facing issues starting the Mirror > > > maker with multiple consumer config files. > > > > > > The original command that we were using with kafka 0.8 was, > > > > > > bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config > > > mirrorCluster1Consumer.properties --consumer.config > > > mirrorCluster2Consumer.properties --consumer.config > > > mirrorCluster3Consumer.properties --consumer.config > > > mirrorCluster4Consumer.properties --producer.config > > > mirrorProducer.properties --whitelist "topic1,topic2" > > > > > > With the current Kafka 0.9 release, the tool throws the following error > > > > > > ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) > > > joptsimple.MultipleArgumentsForOptionException: Found multiple > arguments > > > for option ['consumer.config'], but you asked for only one > > > at joptsimple.OptionSet.valueOf(OptionSet.java:177) > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > Exception in thread "main" java.lang.NullPointerException > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > So I tried setting the --num.streams=4 option in the command, and it > > still > > > throws the same error, next I tried to use the --new.consumer option > but > > > the tool fails again with the same error. > > > > > > Not sure how to get the Mirrormaker tool running with multiple consumer > > > config files. > > > > > > Could not find much documentation regarding this except for the one in > > the > > > kafka documentation ( > > > http://kafka.apache.org/documentation.html#basic_ops_mirror_maker). > > > > > > ps : I was unable to post this in the confluent mailing list and hence > > > posting here. > > > > > > Thanks, > > > Meghana > > > > > > > > > > > -- > > Thanks, > > Ewen > > > > > > -- > -- Guozhang > -- *Meghana Narasimhan* *Software Development DevOps Engineer* *o* 919.827.4318 *e *mnarasim...@bandwidth.com
