RE: Mirror multi-embedded consumer's configuration
Hi, It is OK now, thanks for your help! Best Regards Johnny -Original Message- From: tao xiao [mailto:xiaotao...@gmail.com] Sent: 2016年10月24日 17:43 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration auto.offset.reset=smallest takes effect only when there is no offset committed prior or offset being requested is out of range. I believe your mirror maker has committed offset therefore auto.offset.reset=smallest wont take effect. You can create a new consumer group and try again On Mon, 24 Oct 2016 at 17:25 ZHU Hua B wrote: Hi, Thanks for your info! I set auto.offset.reset=smallest in consumer.properties on target cluster and retest, it still failed. Seems the topic with original messages could not mirror to target cluster while running mirror maker first time, unless produce a new message to this topic to trigger mirror. Thanks! Best Regards Johnny -Original Message- From: tao xiao [mailto:xiaotao...@gmail.com] Sent: 2016年10月24日 17:10 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration You need to set auto.offset.reset=smallest to mirror data from beginning On Mon, 24 Oct 2016 at 17:07 ZHU Hua B wrote: > Hi, > > > Thanks for your info! > > > Before I launch mirror maker first time, there is a topic include some > messages, which have been produced and consumed on source Kafka cluster. > Then I launch mirror maker first time on target cluster, but this > topic did not mirror from source cluster. > > So I keep mirror maker running on target cluster, and produced a new > message to this topic on source cluster, this time the topic mirror > from source cluster to target, which include the new message only, if > this behavior is correct? Thanks! > > > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 16:56 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > Yes, it should mirror again. To mirror the topic from beginning, you > may have to start with different consumer group, > > On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B > > wrote: > > > Hi, > > > > > > Many thanks for your confirm! > > > > I have another question, if I deleted a mirrored topic on the target > > Kafka cluster, if Kafka mirror maker could mirror the same topic > > again from source cluster when I launch mirror maker next time? Thanks! > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > > > -Original Message- > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > Sent: 2016年10月24日 15:03 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > I think we need to update the docs. This behavior got changed > > longtime > ago. > > we need to pass only one consumer configuration. This config will be > > applicable for all the consumer threads/instances. > > > > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > Because from some Kafka wiki I saw "At minimum, the mirror maker > > > takes one or more consumer configurations, a producer > > > configuration and either a whitelist or a blacklist", but the test > > > failed, so I want to know if Kafka mirror really support more > > > consumer configurations at > > minimum? Thanks! > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > > > > -Original Message- > > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > > Sent: 2016年10月24日 13:48 > > > To: users@kafka.apache.org > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > why are you passing "consumer.config" twice? > > > > > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Johnny > > > > > > > > -Original Message- > > > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > > > Sent: 2016年10月2
RE: Mirror multi-embedded consumer's configuration
Yes. //h...@confluent.io Original message From: ZHU Hua B Date: 10/24/16 12:09 AM (GMT-08:00) To: users@kafka.apache.org Subject: RE: Mirror multi-embedded consumer's configuration Hi, Many thanks for your confirm! I have another question, if I deleted a mirrored topic on the target Kafka cluster, if Kafka mirror maker could mirror the same topic again from source cluster when I launch mirror maker next time? Thanks! Best Regards Johnny -Original Message- From: Manikumar [mailto:manikumar.re...@gmail.com] Sent: 2016年10月24日 15:03 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration I think we need to update the docs. This behavior got changed longtime ago. we need to pass only one consumer configuration. This config will be applicable for all the consumer threads/instances. On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B wrote: > Hi, > > > Because from some Kafka wiki I saw "At minimum, the mirror maker takes > one or more consumer configurations, a producer configuration and > either a whitelist or a blacklist", but the test failed, so I want to > know if Kafka mirror really support more consumer configurations at minimum? > Thanks! > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 13:48 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > why are you passing "consumer.config" twice? > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > > wrote: > > > Hi, > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > -Original Message- > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > Sent: 2016年10月24日 12:22 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > Which version are you using for the MM? > > > > Guozhang > > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > Anybody could help to answer below question? Thanks! > > > > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > From: ZHU Hua B > > > Sent: 2016年10月19日 16:22 > > > To: 'users@kafka.apache.org' > > > Subject: Mirror multi-embedded consumer's configuration > > > > > > Hi, > > > > > > > > > I launch Kafka mirror maker with multi-embedded consumer's > > > configuration but failed as below, what’s the mean of “you asked > > > for only one”, is there an option control it? Thanks! > > > > > > # bin/kafka-mirror-maker.sh --consumer.config > > > config/consumer-1.properties --consumer.config > > > config/consumer-2.properties --num.streams 2 --producer.config > > config/producer.properties --whitelist '.*' > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > > > (kafka.tools.MirrorMaker$) > > > joptsimple.MultipleArgumentsForOptionException: Found multiple > > > arguments for option consumer.config, but you asked for only one > > > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > Exception in thread "main" java.lang.NullPointerException > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > > > > > > > -- > > -- Guozhang > > >
Re: Mirror multi-embedded consumer's configuration
auto.offset.reset=smallest takes effect only when there is no offset committed prior or offset being requested is out of range. I believe your mirror maker has committed offset therefore auto.offset.reset=smallest wont take effect. You can create a new consumer group and try again On Mon, 24 Oct 2016 at 17:25 ZHU Hua B wrote: Hi, Thanks for your info! I set auto.offset.reset=smallest in consumer.properties on target cluster and retest, it still failed. Seems the topic with original messages could not mirror to target cluster while running mirror maker first time, unless produce a new message to this topic to trigger mirror. Thanks! Best Regards Johnny -Original Message- From: tao xiao [mailto:xiaotao...@gmail.com] Sent: 2016年10月24日 17:10 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration You need to set auto.offset.reset=smallest to mirror data from beginning On Mon, 24 Oct 2016 at 17:07 ZHU Hua B wrote: > Hi, > > > Thanks for your info! > > > Before I launch mirror maker first time, there is a topic include some > messages, which have been produced and consumed on source Kafka cluster. > Then I launch mirror maker first time on target cluster, but this > topic did not mirror from source cluster. > > So I keep mirror maker running on target cluster, and produced a new > message to this topic on source cluster, this time the topic mirror > from source cluster to target, which include the new message only, if > this behavior is correct? Thanks! > > > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 16:56 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > Yes, it should mirror again. To mirror the topic from beginning, you > may have to start with different consumer group, > > On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B > > wrote: > > > Hi, > > > > > > Many thanks for your confirm! > > > > I have another question, if I deleted a mirrored topic on the target > > Kafka cluster, if Kafka mirror maker could mirror the same topic > > again from source cluster when I launch mirror maker next time? Thanks! > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > > > -Original Message- > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > Sent: 2016年10月24日 15:03 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > I think we need to update the docs. This behavior got changed > > longtime > ago. > > we need to pass only one consumer configuration. This config will be > > applicable for all the consumer threads/instances. > > > > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > Because from some Kafka wiki I saw "At minimum, the mirror maker > > > takes one or more consumer configurations, a producer > > > configuration and either a whitelist or a blacklist", but the test > > > failed, so I want to know if Kafka mirror really support more > > > consumer configurations at > > minimum? Thanks! > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > > > > -Original Message- > > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > > Sent: 2016年10月24日 13:48 > > > To: users@kafka.apache.org > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > why are you passing "consumer.config" twice? > > > > > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Johnny > > > > > > > > -Original Message- > > > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > > > Sent: 2016年10月24日 12:22 > > > > To: users@kafka.apache.org > > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > > > Which version are you using for the MM? > > > > > > > > Guozhang > > >
RE: Mirror multi-embedded consumer's configuration
Hi, Thanks for your info! I set auto.offset.reset=smallest in consumer.properties on target cluster and retest, it still failed. Seems the topic with original messages could not mirror to target cluster while running mirror maker first time, unless produce a new message to this topic to trigger mirror. Thanks! Best Regards Johnny -Original Message- From: tao xiao [mailto:xiaotao...@gmail.com] Sent: 2016年10月24日 17:10 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration You need to set auto.offset.reset=smallest to mirror data from beginning On Mon, 24 Oct 2016 at 17:07 ZHU Hua B wrote: > Hi, > > > Thanks for your info! > > > Before I launch mirror maker first time, there is a topic include some > messages, which have been produced and consumed on source Kafka cluster. > Then I launch mirror maker first time on target cluster, but this > topic did not mirror from source cluster. > > So I keep mirror maker running on target cluster, and produced a new > message to this topic on source cluster, this time the topic mirror > from source cluster to target, which include the new message only, if > this behavior is correct? Thanks! > > > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 16:56 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > Yes, it should mirror again. To mirror the topic from beginning, you > may have to start with different consumer group, > > On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B > > wrote: > > > Hi, > > > > > > Many thanks for your confirm! > > > > I have another question, if I deleted a mirrored topic on the target > > Kafka cluster, if Kafka mirror maker could mirror the same topic > > again from source cluster when I launch mirror maker next time? Thanks! > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > > > -Original Message- > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > Sent: 2016年10月24日 15:03 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > I think we need to update the docs. This behavior got changed > > longtime > ago. > > we need to pass only one consumer configuration. This config will be > > applicable for all the consumer threads/instances. > > > > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > Because from some Kafka wiki I saw "At minimum, the mirror maker > > > takes one or more consumer configurations, a producer > > > configuration and either a whitelist or a blacklist", but the test > > > failed, so I want to know if Kafka mirror really support more > > > consumer configurations at > > minimum? Thanks! > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > > > > -Original Message- > > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > > Sent: 2016年10月24日 13:48 > > > To: users@kafka.apache.org > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > why are you passing "consumer.config" twice? > > > > > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Johnny > > > > > > > > -Original Message- > > > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > > > Sent: 2016年10月24日 12:22 > > > > To: users@kafka.apache.org > > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > > > Which version are you using for the MM? > > > > > > > > Guozhang > > > > > > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > Anybody could help to answer below question? Thanks! > > >
Re: Mirror multi-embedded consumer's configuration
You need to set auto.offset.reset=smallest to mirror data from beginning On Mon, 24 Oct 2016 at 17:07 ZHU Hua B wrote: > Hi, > > > Thanks for your info! > > > Before I launch mirror maker first time, there is a topic include some > messages, which have been produced and consumed on source Kafka cluster. > Then I launch mirror maker first time on target cluster, but this topic did > not mirror from source cluster. > > So I keep mirror maker running on target cluster, and produced a new > message to this topic on source cluster, this time the topic mirror from > source cluster to target, which include the new message only, if this > behavior is correct? Thanks! > > > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 16:56 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > Yes, it should mirror again. To mirror the topic from beginning, you may > have to start with different consumer group, > > On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B > wrote: > > > Hi, > > > > > > Many thanks for your confirm! > > > > I have another question, if I deleted a mirrored topic on the target > > Kafka cluster, if Kafka mirror maker could mirror the same topic again > > from source cluster when I launch mirror maker next time? Thanks! > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > > > -Original Message- > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > Sent: 2016年10月24日 15:03 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > I think we need to update the docs. This behavior got changed longtime > ago. > > we need to pass only one consumer configuration. This config will be > > applicable for all the consumer threads/instances. > > > > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > Because from some Kafka wiki I saw "At minimum, the mirror maker > > > takes one or more consumer configurations, a producer configuration > > > and either a whitelist or a blacklist", but the test failed, so I > > > want to know if Kafka mirror really support more consumer > > > configurations at > > minimum? Thanks! > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > > > > -Original Message- > > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > > Sent: 2016年10月24日 13:48 > > > To: users@kafka.apache.org > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > why are you passing "consumer.config" twice? > > > > > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Johnny > > > > > > > > -Original Message- > > > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > > > Sent: 2016年10月24日 12:22 > > > > To: users@kafka.apache.org > > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > > > Which version are you using for the MM? > > > > > > > > Guozhang > > > > > > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > Anybody could help to answer below question? Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > Johnny > > > > > > > > > > From: ZHU Hua B > > > > > Sent: 2016年10月19日 16:22 > > > > > To: 'users@kafka.apache.org' > > > > > Subject: Mirror multi-embedded consumer's configuration > > > > > > > > > > Hi, > > > >
RE: Mirror multi-embedded consumer's configuration
Hi, Thanks for your info! Before I launch mirror maker first time, there is a topic include some messages, which have been produced and consumed on source Kafka cluster. Then I launch mirror maker first time on target cluster, but this topic did not mirror from source cluster. So I keep mirror maker running on target cluster, and produced a new message to this topic on source cluster, this time the topic mirror from source cluster to target, which include the new message only, if this behavior is correct? Thanks! Best Regards Johnny -Original Message- From: Manikumar [mailto:manikumar.re...@gmail.com] Sent: 2016年10月24日 16:56 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration Yes, it should mirror again. To mirror the topic from beginning, you may have to start with different consumer group, On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B wrote: > Hi, > > > Many thanks for your confirm! > > I have another question, if I deleted a mirrored topic on the target > Kafka cluster, if Kafka mirror maker could mirror the same topic again > from source cluster when I launch mirror maker next time? Thanks! > > > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 15:03 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > I think we need to update the docs. This behavior got changed longtime ago. > we need to pass only one consumer configuration. This config will be > applicable for all the consumer threads/instances. > > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B > > wrote: > > > Hi, > > > > > > Because from some Kafka wiki I saw "At minimum, the mirror maker > > takes one or more consumer configurations, a producer configuration > > and either a whitelist or a blacklist", but the test failed, so I > > want to know if Kafka mirror really support more consumer > > configurations at > minimum? Thanks! > > > > > > > > > > Best Regards > > > > Johnny > > > > > > -Original Message- > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > Sent: 2016年10月24日 13:48 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > why are you passing "consumer.config" twice? > > > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > -Original Message- > > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > > Sent: 2016年10月24日 12:22 > > > To: users@kafka.apache.org > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > Which version are you using for the MM? > > > > > > Guozhang > > > > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > Anybody could help to answer below question? Thanks! > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Johnny > > > > > > > > From: ZHU Hua B > > > > Sent: 2016年10月19日 16:22 > > > > To: 'users@kafka.apache.org' > > > > Subject: Mirror multi-embedded consumer's configuration > > > > > > > > Hi, > > > > > > > > > > > > I launch Kafka mirror maker with multi-embedded consumer's > > > > configuration but failed as below, what’s the mean of “you asked > > > > for only one”, is there an option control it? Thanks! > > > > > > > > # bin/kafka-mirror-maker.sh --consumer.config > > > > config/consumer-1.properties --consumer.config > > > > config/consumer-2.properties --num.streams 2 --producer.config > > > config/producer.properties --whitelist '.*' > > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > > > > (kafka.tools.MirrorMaker$) > > > > joptsimple.MultipleArgumentsForOptionException: Found multiple > > > > arguments for option consumer.config, but you asked for only one > > > > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > Exception in thread "main" java.lang.NullPointerException > > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Johnny > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > >
Re: Mirror multi-embedded consumer's configuration
Yes, it should mirror again. To mirror the topic from beginning, you may have to start with different consumer group, On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B wrote: > Hi, > > > Many thanks for your confirm! > > I have another question, if I deleted a mirrored topic on the target Kafka > cluster, if Kafka mirror maker could mirror the same topic again from > source cluster when I launch mirror maker next time? Thanks! > > > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 15:03 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > I think we need to update the docs. This behavior got changed longtime ago. > we need to pass only one consumer configuration. This config will be > applicable for all the consumer threads/instances. > > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B > wrote: > > > Hi, > > > > > > Because from some Kafka wiki I saw "At minimum, the mirror maker takes > > one or more consumer configurations, a producer configuration and > > either a whitelist or a blacklist", but the test failed, so I want to > > know if Kafka mirror really support more consumer configurations at > minimum? Thanks! > > > > > > > > > > Best Regards > > > > Johnny > > > > > > -Original Message- > > From: Manikumar [mailto:manikumar.re...@gmail.com] > > Sent: 2016年10月24日 13:48 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > why are you passing "consumer.config" twice? > > > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > -Original Message- > > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > > Sent: 2016年10月24日 12:22 > > > To: users@kafka.apache.org > > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > > > Which version are you using for the MM? > > > > > > Guozhang > > > > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > Anybody could help to answer below question? Thanks! > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Johnny > > > > > > > > From: ZHU Hua B > > > > Sent: 2016年10月19日 16:22 > > > > To: 'users@kafka.apache.org' > > > > Subject: Mirror multi-embedded consumer's configuration > > > > > > > > Hi, > > > > > > > > > > > > I launch Kafka mirror maker with multi-embedded consumer's > > > > configuration but failed as below, what’s the mean of “you asked > > > > for only one”, is there an option control it? Thanks! > > > > > > > > # bin/kafka-mirror-maker.sh --consumer.config > > > > config/consumer-1.properties --consumer.config > > > > config/consumer-2.properties --num.streams 2 --producer.config > > > config/producer.properties --whitelist '.*' > > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > > > > (kafka.tools.MirrorMaker$) > > > > joptsimple.MultipleArgumentsForOptionException: Found multiple > > > > arguments for option consumer.config, but you asked for only one > > > > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > Exception in thread "main" java.lang.NullPointerException > > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > Johnny > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > >
RE: Mirror multi-embedded consumer's configuration
Hi, Many thanks for your confirm! I have another question, if I deleted a mirrored topic on the target Kafka cluster, if Kafka mirror maker could mirror the same topic again from source cluster when I launch mirror maker next time? Thanks! Best Regards Johnny -Original Message- From: Manikumar [mailto:manikumar.re...@gmail.com] Sent: 2016年10月24日 15:03 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration I think we need to update the docs. This behavior got changed longtime ago. we need to pass only one consumer configuration. This config will be applicable for all the consumer threads/instances. On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B wrote: > Hi, > > > Because from some Kafka wiki I saw "At minimum, the mirror maker takes > one or more consumer configurations, a producer configuration and > either a whitelist or a blacklist", but the test failed, so I want to > know if Kafka mirror really support more consumer configurations at minimum? > Thanks! > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 13:48 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > why are you passing "consumer.config" twice? > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > > wrote: > > > Hi, > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > -Original Message- > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > Sent: 2016年10月24日 12:22 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > Which version are you using for the MM? > > > > Guozhang > > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > Anybody could help to answer below question? Thanks! > > > > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > From: ZHU Hua B > > > Sent: 2016年10月19日 16:22 > > > To: 'users@kafka.apache.org' > > > Subject: Mirror multi-embedded consumer's configuration > > > > > > Hi, > > > > > > > > > I launch Kafka mirror maker with multi-embedded consumer's > > > configuration but failed as below, what’s the mean of “you asked > > > for only one”, is there an option control it? Thanks! > > > > > > # bin/kafka-mirror-maker.sh --consumer.config > > > config/consumer-1.properties --consumer.config > > > config/consumer-2.properties --num.streams 2 --producer.config > > config/producer.properties --whitelist '.*' > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > > > (kafka.tools.MirrorMaker$) > > > joptsimple.MultipleArgumentsForOptionException: Found multiple > > > arguments for option consumer.config, but you asked for only one > > > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > Exception in thread "main" java.lang.NullPointerException > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > > > > > > > -- > > -- Guozhang > > >
Re: Mirror multi-embedded consumer's configuration
I think we need to update the docs. This behavior got changed longtime ago. we need to pass only one consumer configuration. This config will be applicable for all the consumer threads/instances. On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B wrote: > Hi, > > > Because from some Kafka wiki I saw "At minimum, the mirror maker takes one > or more consumer configurations, a producer configuration and either a > whitelist or a blacklist", but the test failed, so I want to know if Kafka > mirror really support more consumer configurations at minimum? Thanks! > > > > > Best Regards > > Johnny > > > -Original Message- > From: Manikumar [mailto:manikumar.re...@gmail.com] > Sent: 2016年10月24日 13:48 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > why are you passing "consumer.config" twice? > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B > wrote: > > > Hi, > > > > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > -Original Message- > > From: Guozhang Wang [mailto:wangg...@gmail.com] > > Sent: 2016年10月24日 12:22 > > To: users@kafka.apache.org > > Subject: Re: Mirror multi-embedded consumer's configuration > > > > Which version are you using for the MM? > > > > Guozhang > > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > > > > wrote: > > > > > Hi, > > > > > > > > > Anybody could help to answer below question? Thanks! > > > > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > From: ZHU Hua B > > > Sent: 2016年10月19日 16:22 > > > To: 'users@kafka.apache.org' > > > Subject: Mirror multi-embedded consumer's configuration > > > > > > Hi, > > > > > > > > > I launch Kafka mirror maker with multi-embedded consumer's > > > configuration but failed as below, what’s the mean of “you asked for > > > only one”, is there an option control it? Thanks! > > > > > > # bin/kafka-mirror-maker.sh --consumer.config > > > config/consumer-1.properties --consumer.config > > > config/consumer-2.properties --num.streams 2 --producer.config > > config/producer.properties --whitelist '.*' > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > > > (kafka.tools.MirrorMaker$) > > > joptsimple.MultipleArgumentsForOptionException: Found multiple > > > arguments for option consumer.config, but you asked for only one > > > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > Exception in thread "main" java.lang.NullPointerException > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > Johnny > > > > > > > > > > > > -- > > -- Guozhang > > >
RE: Mirror multi-embedded consumer's configuration
Hi, Because from some Kafka wiki I saw "At minimum, the mirror maker takes one or more consumer configurations, a producer configuration and either a whitelist or a blacklist", but the test failed, so I want to know if Kafka mirror really support more consumer configurations at minimum? Thanks! Best Regards Johnny -Original Message- From: Manikumar [mailto:manikumar.re...@gmail.com] Sent: 2016年10月24日 13:48 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration why are you passing "consumer.config" twice? On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B wrote: > Hi, > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > Best Regards > > Johnny > > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: 2016年10月24日 12:22 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > Which version are you using for the MM? > > Guozhang > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > > wrote: > > > Hi, > > > > > > Anybody could help to answer below question? Thanks! > > > > > > > > > > > > Best Regards > > > > Johnny > > > > From: ZHU Hua B > > Sent: 2016年10月19日 16:22 > > To: 'users@kafka.apache.org' > > Subject: Mirror multi-embedded consumer's configuration > > > > Hi, > > > > > > I launch Kafka mirror maker with multi-embedded consumer's > > configuration but failed as below, what’s the mean of “you asked for > > only one”, is there an option control it? Thanks! > > > > # bin/kafka-mirror-maker.sh --consumer.config > > config/consumer-1.properties --consumer.config > > config/consumer-2.properties --num.streams 2 --producer.config > config/producer.properties --whitelist '.*' > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > > (kafka.tools.MirrorMaker$) > > joptsimple.MultipleArgumentsForOptionException: Found multiple > > arguments for option consumer.config, but you asked for only one > > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > Exception in thread "main" java.lang.NullPointerException > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > > > > -- > -- Guozhang >
Re: Mirror multi-embedded consumer's configuration
why are you passing "consumer.config" twice? On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B wrote: > Hi, > > > The version of Kafka I used is 0.10.0.0. Thanks! > > > > > > > Best Regards > > Johnny > > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: 2016年10月24日 12:22 > To: users@kafka.apache.org > Subject: Re: Mirror multi-embedded consumer's configuration > > Which version are you using for the MM? > > Guozhang > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B > wrote: > > > Hi, > > > > > > Anybody could help to answer below question? Thanks! > > > > > > > > > > > > Best Regards > > > > Johnny > > > > From: ZHU Hua B > > Sent: 2016年10月19日 16:22 > > To: 'users@kafka.apache.org' > > Subject: Mirror multi-embedded consumer's configuration > > > > Hi, > > > > > > I launch Kafka mirror maker with multi-embedded consumer's > > configuration but failed as below, what’s the mean of “you asked for > > only one”, is there an option control it? Thanks! > > > > # bin/kafka-mirror-maker.sh --consumer.config > > config/consumer-1.properties --consumer.config > > config/consumer-2.properties --num.streams 2 --producer.config > config/producer.properties --whitelist '.*' > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > > (kafka.tools.MirrorMaker$) > > joptsimple.MultipleArgumentsForOptionException: Found multiple > > arguments for option consumer.config, but you asked for only one > > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > Exception in thread "main" java.lang.NullPointerException > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > > > > > > > > Best Regards > > > > Johnny > > > > > > > -- > -- Guozhang >
RE: Mirror multi-embedded consumer's configuration
Hi, The version of Kafka I used is 0.10.0.0. Thanks! Best Regards Johnny -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: 2016年10月24日 12:22 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration Which version are you using for the MM? Guozhang On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B wrote: > Hi, > > > Anybody could help to answer below question? Thanks! > > > > > > Best Regards > > Johnny > > From: ZHU Hua B > Sent: 2016年10月19日 16:22 > To: 'users@kafka.apache.org' > Subject: Mirror multi-embedded consumer's configuration > > Hi, > > > I launch Kafka mirror maker with multi-embedded consumer's > configuration but failed as below, what’s the mean of “you asked for > only one”, is there an option control it? Thanks! > > # bin/kafka-mirror-maker.sh --consumer.config > config/consumer-1.properties --consumer.config > config/consumer-2.properties --num.streams 2 --producer.config > config/producer.properties --whitelist '.*' > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > (kafka.tools.MirrorMaker$) > joptsimple.MultipleArgumentsForOptionException: Found multiple > arguments for option consumer.config, but you asked for only one > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > Exception in thread "main" java.lang.NullPointerException > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > Best Regards > > Johnny > > -- -- Guozhang
Re: Mirror multi-embedded consumer's configuration
Which version are you using for the MM? Guozhang On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B wrote: > Hi, > > > Anybody could help to answer below question? Thanks! > > > > > > Best Regards > > Johnny > > From: ZHU Hua B > Sent: 2016年10月19日 16:22 > To: 'users@kafka.apache.org' > Subject: Mirror multi-embedded consumer's configuration > > Hi, > > > I launch Kafka mirror maker with multi-embedded consumer's configuration > but failed as below, what’s the mean of “you asked for only one”, is there > an option control it? Thanks! > > # bin/kafka-mirror-maker.sh --consumer.config config/consumer-1.properties > --consumer.config config/consumer-2.properties --num.streams 2 > --producer.config config/producer.properties --whitelist '.*' > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. > (kafka.tools.MirrorMaker$) > joptsimple.MultipleArgumentsForOptionException: Found multiple arguments > for option consumer.config, but you asked for only one > at joptsimple.OptionSet.valueOf(OptionSet.java:179) > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > Exception in thread "main" java.lang.NullPointerException > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) > at kafka.tools.MirrorMaker.main(MirrorMaker.scala) > > > > > > > Best Regards > > Johnny > > -- -- Guozhang
RE: Mirror multi-embedded consumer's configuration
Hi, Anybody could help to answer below question? Thanks! Best Regards Johnny From: ZHU Hua B Sent: 2016年10月19日 16:22 To: 'users@kafka.apache.org' Subject: Mirror multi-embedded consumer's configuration Hi, I launch Kafka mirror maker with multi-embedded consumer's configuration but failed as below, what’s the mean of “you asked for only one”, is there an option control it? Thanks! # bin/kafka-mirror-maker.sh --consumer.config config/consumer-1.properties --consumer.config config/consumer-2.properties --num.streams 2 --producer.config config/producer.properties --whitelist '.*' [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) joptsimple.MultipleArgumentsForOptionException: Found multiple arguments for option consumer.config, but you asked for only one at joptsimple.OptionSet.valueOf(OptionSet.java:179) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Exception in thread "main" java.lang.NullPointerException at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Best Regards Johnny
RE: Mirror multi-embedded consumer's configuration
Hi, Anybody could help to answer below question? Thanks! Best Regards Johnny From: ZHU Hua B Sent: 2016年10月19日 16:22 To: 'users@kafka.apache.org' Subject: Mirror multi-embedded consumer's configuration Hi, I launch Kafka mirror maker with multi-embedded consumer's configuration but failed as below, what’s the mean of “you asked for only one”, is there an option control it? Thanks! # bin/kafka-mirror-maker.sh --consumer.config config/consumer-1.properties --consumer.config config/consumer-2.properties --num.streams 2 --producer.config config/producer.properties --whitelist '.*' [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) joptsimple.MultipleArgumentsForOptionException: Found multiple arguments for option consumer.config, but you asked for only one at joptsimple.OptionSet.valueOf(OptionSet.java:179) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Exception in thread "main" java.lang.NullPointerException at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Best Regards Johnny
Mirror multi-embedded consumer's configuration
Hi, I launch Kafka mirror maker with multi-embedded consumer's configuration but failed as below, what's the mean of "you asked for only one", is there an option control it? Thanks! # bin/kafka-mirror-maker.sh --consumer.config config/consumer-1.properties --consumer.config config/consumer-2.properties --num.streams 2 --producer.config config/producer.properties --whitelist '.*' [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker. (kafka.tools.MirrorMaker$) joptsimple.MultipleArgumentsForOptionException: Found multiple arguments for option consumer.config, but you asked for only one at joptsimple.OptionSet.valueOf(OptionSet.java:179) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Exception in thread "main" java.lang.NullPointerException at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Best Regards Johnny
