Can you attach your mirror maker log? On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <kumarsharma.ni...@gmail.com> wrote:
> i tried with auto.offset.reset=smallest, but still not working.. > > there is data in my source cluster > > Regards, > Nitin Kumar Sharma. > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xiaotao...@gmail.com> wrote: > > > Do you have data sending to *testtopic? *By default mirror maker only > > consumes data being sent after it taps into the topic. you need to keep > > sending data to the topic after mirror maker connection is established. > If > > you want to change the behavior you can set auto.offset.reset=smallest so > > that any new mirror maker coming to the topic will start from the > smallest > > offset > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma < > kumarsharma.ni...@gmail.com > > > > > wrote: > > > > > thanks Xiao > > > > > > I tried MirrorMaker option in my test environment but failed. I am not > > > able to see the log getting copied to destination cluster. I see in the > > log > > > of MirrorMaker process that connection is successfully established > > between > > > source and destination cluster but still not sure what i causing the > > problem > > > > > > Env. Setup ==> > > > > > > I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) > and > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03). > > > Destination Clustern (Qenv05) -- i have 2 broker > (Qenv05kf01,Qenv05kf02) > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03). > > > > > > II). i have kept consumer and producer properties file in one of the > > > source kafka broker config folder. > > > > > > III).i have executed following command from the same kafka broker to > > > start the process.. log are attached : > > > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker > > -consumer.config > > > /app/kafka/config/consumer1.properties --num.streams=2 > --producer.config > > > /app/kafka/config/producer1.properties --whitelist testtopic > > > > > > > > > IV). I tried Consumer offset tracker tool also, while Mirror Maker > > running > > > . I tried by launching second session of same broker where mirror maker > > is > > > running. I got error message that "*NoNode for > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached. > > > > > > > > > Regards, > > > Nitin Kumar Sharma. > > > > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xiaotao...@gmail.com> > wrote: > > > > > >> Both consumer-1 and consumer-2 are properties of source clusters > mirror > > >> maker transfers data from. Mirror maker is designed to be able to > > consume > > >> data from N sources (N >= 1) and transfer data to one destination > > cluster. > > >> You are free to supply as many consumer properties as you want to > > instruct > > >> mirror maker where to consumer data from. > > >> > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma < > > >> kumarsharma.ni...@gmail.com> > > >> wrote: > > >> > > >> > thanks Mayuresh and Jiangjie for your response. > > >> > > > >> > I have actually not understood Mirror maker clearly and hence bit > > >> skeptical > > >> > if i will be able to execute it effectively. > > >> > > > >> > Online i hv seen the following command to be execute, but not > > understood > > >> > what is consumer-1 & -2.properties here? do i need to copy from my > > >> > consumer code? also, any reason why to provide consumer property? > > >> > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config > > >> > consumer-1.properties --consumer.config consumer-2.properties > > >> > --producer.config producer.properties --whitelist my-topic > > >> > > > >> > > > >> > Regards, > > >> > Nitin Kumar Sharma. > > >> > > > >> > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat < > > >> > gharatmayures...@gmail.com > > >> > > wrote: > > >> > > > >> > > You can use the Mirror maker to move data from one data center to > > >> other > > >> > and > > >> > > once all the data has been moved you can shut down the source data > > >> center > > >> > > by doing a controlled shutdown. > > >> > > > > >> > > Thanks, > > >> > > > > >> > > Mayuresh > > >> > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin > > >> <j...@linkedin.com.invalid > > >> > > > > >> > > wrote: > > >> > > > > >> > > > If you want to do a seamless migration. I think a better way is > to > > >> > build > > >> > > a > > >> > > > cross datacenter Kafka cluster temporarily. So the process is: > > >> > > > 1. Add several new Kafka brokers in your new datacenter and add > > >> them to > > >> > > > the old cluster. > > >> > > > 2. Use replica assignment tool to reassign all the partitions to > > >> > brokers > > >> > > > in new datacenter. > > >> > > > 3. Perform controlled shutdown on the brokers in old datacenter. > > >> > > > > > >> > > > Jiangjie (Becket) Qin > > >> > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" < > kumarsharma.ni...@gmail.com> > > >> > wrote: > > >> > > > > > >> > > > >Hi Team, > > >> > > > > > > >> > > > >in my project, we have built a new datacenter for Kafka brokers > > and > > >> > > wants > > >> > > > >to migrate from current datacenter to new one. > > >> > > > > > > >> > > > >Switching producers and consumers wont be a problem provided > New > > >> > > > >Datacenter > > >> > > > >has all the messages of existing Datacenter. > > >> > > > > > > >> > > > > > > >> > > > >i have only 1 topic with 2 partition that need to be > migrated... > > >> that > > >> > > too > > >> > > > >it is only 1 time activity. > > >> > > > > > > >> > > > >Kindly suggest the best way to deal with this situation. > > >> > > > > > > >> > > > > > > >> > > > >Regards, > > >> > > > >Nitin Kumar Sharma. > > >> > > > > > >> > > > > > >> > > > > >> > > > > >> > > -- > > >> > > -Regards, > > >> > > Mayuresh R. Gharat > > >> > > (862) 250-7125 > > >> > > > > >> > > > >> > > >> > > >> > > >> -- > > >> Regards, > > >> Tao > > >> > > > > > > > > > > > > -- > > Regards, > > Tao > > > -- Regards, Tao
