Re: Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
On Fri, Dec 11, 2015 at 3:31 AM, Guozhang Wang wrote: > As Ewen said in KAFKA-1650 we removed the support of multiple consumer > configs in MM, so that you cannot get one MM instance to pull from multiple > origin clusters, but instead you need to use one MM instance for each > origin. > Seems like we should mention this in the upgrade notes. Best, Ismael
Re: Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Thanks Ewen and Guozhang for the details ! On Thu, Dec 10, 2015 at 10:31 PM, Guozhang Wang wrote: > Meghana, > > As Ewen said in KAFKA-1650 we removed the support of multiple consumer > configs in MM, so that you cannot get one MM instance to pull from multiple > origin clusters, but instead you need to use one MM instance for each > origin. > > The motivation for dropping this support is mainly operational: with the > previous settings if one of the original cluster falls into a bad state, we > cannot stop / debug that MM since it also pulls from other clusters. And > other minor issues include metics reporting, etc. For example at LI people > use one MM instance for each origin cluster, that are co-located with the > destination cluster, so if you need to mirror from cluster A / B / C to D, > you will have a A->D / B->D / C->D MM. Moving forward we are also trying to > add exact mirroring features into MM, which would better be a 1-to-1 > mapping. > > Guozhang > > > On Thu, Dec 10, 2015 at 3:21 PM, Ewen Cheslack-Postava > wrote: > > > Meghana, > > > > It looks like this functionality was removed in > > https://issues.apache.org/jira/browse/KAFKA-1650, although I don't see > > explicit discussion of the issue in the JIRA so I'm not sure of the exact > > motivation. Maybe Becket or Guozhang can offer some insight about if it > is > > necessary (that JIRA fixes a data loss issue) or an accidental > > incompatibility. > > > > -Ewen > > > > On Thu, Dec 10, 2015 at 2:32 PM, Meghana Narasimhan < > > mnarasim...@bandwidth.com> wrote: > > > > > Hi, > > > > > > We upgraded to the Confluent 2.0 platform today (we were earlier using > > the > > > Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 > that > > > is packaged with the platform we are facing issues starting the Mirror > > > maker with multiple consumer config files. > > > > > > The original command that we were using with kafka 0.8 was, > > > > > > bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config > > > mirrorCluster1Consumer.properties --consumer.config > > > mirrorCluster2Consumer.properties --consumer.config > > > mirrorCluster3Consumer.properties --consumer.config > > > mirrorCluster4Consumer.properties --producer.config > > > mirrorProducer.properties --whitelist "topic1,topic2" > > > > > > With the current Kafka 0.9 release, the tool throws the following error > > > > > > ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) > > > joptsimple.MultipleArgumentsForOptionException: Found multiple > arguments > > > for option ['consumer.config'], but you asked for only one > > > at joptsimple.OptionSet.valueOf(OptionSet.java:177) > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > Exception in thread "main" java.lang.NullPointerException > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > So I tried setting the --num.streams=4 option in the command, and it > > still > > > throws the same error, next I tried to use the --new.consumer option > but > > > the tool fails again with the same error. > > > > > > Not sure how to get the Mirrormaker tool running with multiple consumer > > > config files. > > > > > > Could not find much documentation regarding this except for the one in > > the > > > kafka documentation ( > > > http://kafka.apache.org/documentation.html#basic_ops_mirror_maker). > > > > > > ps : I was unable to post this in the confluent mailing list and hence > > > posting here. > > > > > > Thanks, > > > Meghana > > > > > > > > > > > -- > > Thanks, > > Ewen > > > > > > -- > -- Guozhang > -- *Meghana Narasimhan* *Software Development DevOps Engineer* *o* 919.827.4318 *e *mnarasim...@bandwidth.com
Re: Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Meghana, As Ewen said in KAFKA-1650 we removed the support of multiple consumer configs in MM, so that you cannot get one MM instance to pull from multiple origin clusters, but instead you need to use one MM instance for each origin. The motivation for dropping this support is mainly operational: with the previous settings if one of the original cluster falls into a bad state, we cannot stop / debug that MM since it also pulls from other clusters. And other minor issues include metics reporting, etc. For example at LI people use one MM instance for each origin cluster, that are co-located with the destination cluster, so if you need to mirror from cluster A / B / C to D, you will have a A->D / B->D / C->D MM. Moving forward we are also trying to add exact mirroring features into MM, which would better be a 1-to-1 mapping. Guozhang On Thu, Dec 10, 2015 at 3:21 PM, Ewen Cheslack-Postava wrote: > Meghana, > > It looks like this functionality was removed in > https://issues.apache.org/jira/browse/KAFKA-1650, although I don't see > explicit discussion of the issue in the JIRA so I'm not sure of the exact > motivation. Maybe Becket or Guozhang can offer some insight about if it is > necessary (that JIRA fixes a data loss issue) or an accidental > incompatibility. > > -Ewen > > On Thu, Dec 10, 2015 at 2:32 PM, Meghana Narasimhan < > mnarasim...@bandwidth.com> wrote: > > > Hi, > > > > We upgraded to the Confluent 2.0 platform today (we were earlier using > the > > Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 that > > is packaged with the platform we are facing issues starting the Mirror > > maker with multiple consumer config files. > > > > The original command that we were using with kafka 0.8 was, > > > > bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config > > mirrorCluster1Consumer.properties --consumer.config > > mirrorCluster2Consumer.properties --consumer.config > > mirrorCluster3Consumer.properties --consumer.config > > mirrorCluster4Consumer.properties --producer.config > > mirrorProducer.properties --whitelist "topic1,topic2" > > > > With the current Kafka 0.9 release, the tool throws the following error > > > > ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) > > joptsimple.MultipleArgumentsForOptionException: Found multiple arguments > > for option ['consumer.config'], but you asked for only one > > at joptsimple.OptionSet.valueOf(OptionSet.java:177) > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225) > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > Exception in thread "main" java.lang.NullPointerException > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276) > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > So I tried setting the --num.streams=4 option in the command, and it > still > > throws the same error, next I tried to use the --new.consumer option but > > the tool fails again with the same error. > > > > Not sure how to get the Mirrormaker tool running with multiple consumer > > config files. > > > > Could not find much documentation regarding this except for the one in > the > > kafka documentation ( > > http://kafka.apache.org/documentation.html#basic_ops_mirror_maker). > > > > ps : I was unable to post this in the confluent mailing list and hence > > posting here. > > > > Thanks, > > Meghana > > > > > > -- > Thanks, > Ewen > -- -- Guozhang
Re: Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Meghana, It looks like this functionality was removed in https://issues.apache.org/jira/browse/KAFKA-1650, although I don't see explicit discussion of the issue in the JIRA so I'm not sure of the exact motivation. Maybe Becket or Guozhang can offer some insight about if it is necessary (that JIRA fixes a data loss issue) or an accidental incompatibility. -Ewen On Thu, Dec 10, 2015 at 2:32 PM, Meghana Narasimhan < mnarasim...@bandwidth.com> wrote: > Hi, > > We upgraded to the Confluent 2.0 platform today (we were earlier using the > Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 that > is packaged with the platform we are facing issues starting the Mirror > maker with multiple consumer config files. > > The original command that we were using with kafka 0.8 was, > > bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config > mirrorCluster1Consumer.properties --consumer.config > mirrorCluster2Consumer.properties --consumer.config > mirrorCluster3Consumer.properties --consumer.config > mirrorCluster4Consumer.properties --producer.config > mirrorProducer.properties --whitelist "topic1,topic2" > > With the current Kafka 0.9 release, the tool throws the following error > > ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) > joptsimple.MultipleArgumentsForOptionException: Found multiple arguments > for option ['consumer.config'], but you asked for only one > at joptsimple.OptionSet.valueOf(OptionSet.java:177) > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225) > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > Exception in thread "main" java.lang.NullPointerException > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276) > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > So I tried setting the --num.streams=4 option in the command, and it still > throws the same error, next I tried to use the --new.consumer option but > the tool fails again with the same error. > > Not sure how to get the Mirrormaker tool running with multiple consumer > config files. > > Could not find much documentation regarding this except for the one in the > kafka documentation ( > http://kafka.apache.org/documentation.html#basic_ops_mirror_maker). > > ps : I was unable to post this in the confluent mailing list and hence > posting here. > > Thanks, > Meghana > -- Thanks, Ewen
Mirrormaker issue with Kafka 0.9 (confluent platform 2.0)
Hi, We upgraded to the Confluent 2.0 platform today (we were earlier using the Kafka 0.8 without the confluent platform). With the latest Kafka 0.9 that is packaged with the platform we are facing issues starting the Mirror maker with multiple consumer config files. The original command that we were using with kafka 0.8 was, bin/kafka-run-class kafka.tools.MirrorMaker --consumer.config mirrorCluster1Consumer.properties --consumer.config mirrorCluster2Consumer.properties --consumer.config mirrorCluster3Consumer.properties --consumer.config mirrorCluster4Consumer.properties --producer.config mirrorProducer.properties --whitelist "topic1,topic2" With the current Kafka 0.9 release, the tool throws the following error ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) joptsimple.MultipleArgumentsForOptionException: Found multiple arguments for option ['consumer.config'], but you asked for only one at joptsimple.OptionSet.valueOf(OptionSet.java:177) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:225) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Exception in thread "main" java.lang.NullPointerException at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:276) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) So I tried setting the --num.streams=4 option in the command, and it still throws the same error, next I tried to use the --new.consumer option but the tool fails again with the same error. Not sure how to get the Mirrormaker tool running with multiple consumer config files. Could not find much documentation regarding this except for the one in the kafka documentation ( http://kafka.apache.org/documentation.html#basic_ops_mirror_maker). ps : I was unable to post this in the confluent mailing list and hence posting here. Thanks, Meghana
