i tried with auto.offset.reset=smallest, but still not working.. there is data in my source cluster
Regards, Nitin Kumar Sharma. On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xiaotao...@gmail.com> wrote: > Do you have data sending to *testtopic? *By default mirror maker only > consumes data being sent after it taps into the topic. you need to keep > sending data to the topic after mirror maker connection is established. If > you want to change the behavior you can set auto.offset.reset=smallest so > that any new mirror maker coming to the topic will start from the smallest > offset > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <kumarsharma.ni...@gmail.com > > > wrote: > > > thanks Xiao > > > > I tried MirrorMaker option in my test environment but failed. I am not > > able to see the log getting copied to destination cluster. I see in the > log > > of MirrorMaker process that connection is successfully established > between > > source and destination cluster but still not sure what i causing the > problem > > > > Env. Setup ==> > > > > I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) and > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03). > > Destination Clustern (Qenv05) -- i have 2 broker (Qenv05kf01,Qenv05kf02) > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03). > > > > II). i have kept consumer and producer properties file in one of the > > source kafka broker config folder. > > > > III).i have executed following command from the same kafka broker to > > start the process.. log are attached : > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker > -consumer.config > > /app/kafka/config/consumer1.properties --num.streams=2 --producer.config > > /app/kafka/config/producer1.properties --whitelist testtopic > > > > > > IV). I tried Consumer offset tracker tool also, while Mirror Maker > running > > . I tried by launching second session of same broker where mirror maker > is > > running. I got error message that "*NoNode for > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached. > > > > > > Regards, > > Nitin Kumar Sharma. > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xiaotao...@gmail.com> wrote: > > > >> Both consumer-1 and consumer-2 are properties of source clusters mirror > >> maker transfers data from. Mirror maker is designed to be able to > consume > >> data from N sources (N >= 1) and transfer data to one destination > cluster. > >> You are free to supply as many consumer properties as you want to > instruct > >> mirror maker where to consumer data from. > >> > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma < > >> kumarsharma.ni...@gmail.com> > >> wrote: > >> > >> > thanks Mayuresh and Jiangjie for your response. > >> > > >> > I have actually not understood Mirror maker clearly and hence bit > >> skeptical > >> > if i will be able to execute it effectively. > >> > > >> > Online i hv seen the following command to be execute, but not > understood > >> > what is consumer-1 & -2.properties here? do i need to copy from my > >> > consumer code? also, any reason why to provide consumer property? > >> > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config > >> > consumer-1.properties --consumer.config consumer-2.properties > >> > --producer.config producer.properties --whitelist my-topic > >> > > >> > > >> > Regards, > >> > Nitin Kumar Sharma. > >> > > >> > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat < > >> > gharatmayures...@gmail.com > >> > > wrote: > >> > > >> > > You can use the Mirror maker to move data from one data center to > >> other > >> > and > >> > > once all the data has been moved you can shut down the source data > >> center > >> > > by doing a controlled shutdown. > >> > > > >> > > Thanks, > >> > > > >> > > Mayuresh > >> > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin > >> <j...@linkedin.com.invalid > >> > > > >> > > wrote: > >> > > > >> > > > If you want to do a seamless migration. I think a better way is to > >> > build > >> > > a > >> > > > cross datacenter Kafka cluster temporarily. So the process is: > >> > > > 1. Add several new Kafka brokers in your new datacenter and add > >> them to > >> > > > the old cluster. > >> > > > 2. Use replica assignment tool to reassign all the partitions to > >> > brokers > >> > > > in new datacenter. > >> > > > 3. Perform controlled shutdown on the brokers in old datacenter. > >> > > > > >> > > > Jiangjie (Becket) Qin > >> > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <kumarsharma.ni...@gmail.com> > >> > wrote: > >> > > > > >> > > > >Hi Team, > >> > > > > > >> > > > >in my project, we have built a new datacenter for Kafka brokers > and > >> > > wants > >> > > > >to migrate from current datacenter to new one. > >> > > > > > >> > > > >Switching producers and consumers wont be a problem provided New > >> > > > >Datacenter > >> > > > >has all the messages of existing Datacenter. > >> > > > > > >> > > > > > >> > > > >i have only 1 topic with 2 partition that need to be migrated... > >> that > >> > > too > >> > > > >it is only 1 time activity. > >> > > > > > >> > > > >Kindly suggest the best way to deal with this situation. > >> > > > > > >> > > > > > >> > > > >Regards, > >> > > > >Nitin Kumar Sharma. > >> > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > -Regards, > >> > > Mayuresh R. Gharat > >> > > (862) 250-7125 > >> > > > >> > > >> > >> > >> > >> -- > >> Regards, > >> Tao > >> > > > > > > > -- > Regards, > Tao >
