Do you have data sending to *testtopic? *By default mirror maker only consumes data being sent after it taps into the topic. you need to keep sending data to the topic after mirror maker connection is established. If you want to change the behavior you can set auto.offset.reset=smallest so that any new mirror maker coming to the topic will start from the smallest offset
On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma <kumarsharma.ni...@gmail.com> wrote: > thanks Xiao > > I tried MirrorMaker option in my test environment but failed. I am not > able to see the log getting copied to destination cluster. I see in the log > of MirrorMaker process that connection is successfully established between > source and destination cluster but still not sure what i causing the problem > > Env. Setup ==> > > I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) and > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03). > Destination Clustern (Qenv05) -- i have 2 broker (Qenv05kf01,Qenv05kf02) > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03). > > II). i have kept consumer and producer properties file in one of the > source kafka broker config folder. > > III).i have executed following command from the same kafka broker to > start the process.. log are attached : > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker -consumer.config > /app/kafka/config/consumer1.properties --num.streams=2 --producer.config > /app/kafka/config/producer1.properties --whitelist testtopic > > > IV). I tried Consumer offset tracker tool also, while Mirror Maker running > . I tried by launching second session of same broker where mirror maker is > running. I got error message that "*NoNode for > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached. > > > Regards, > Nitin Kumar Sharma. > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xiaotao...@gmail.com> wrote: > >> Both consumer-1 and consumer-2 are properties of source clusters mirror >> maker transfers data from. Mirror maker is designed to be able to consume >> data from N sources (N >= 1) and transfer data to one destination cluster. >> You are free to supply as many consumer properties as you want to instruct >> mirror maker where to consumer data from. >> >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma < >> kumarsharma.ni...@gmail.com> >> wrote: >> >> > thanks Mayuresh and Jiangjie for your response. >> > >> > I have actually not understood Mirror maker clearly and hence bit >> skeptical >> > if i will be able to execute it effectively. >> > >> > Online i hv seen the following command to be execute, but not understood >> > what is consumer-1 & -2.properties here? do i need to copy from my >> > consumer code? also, any reason why to provide consumer property? >> > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config >> > consumer-1.properties --consumer.config consumer-2.properties >> > --producer.config producer.properties --whitelist my-topic >> > >> > >> > Regards, >> > Nitin Kumar Sharma. >> > >> > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat < >> > gharatmayures...@gmail.com >> > > wrote: >> > >> > > You can use the Mirror maker to move data from one data center to >> other >> > and >> > > once all the data has been moved you can shut down the source data >> center >> > > by doing a controlled shutdown. >> > > >> > > Thanks, >> > > >> > > Mayuresh >> > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin >> <j...@linkedin.com.invalid >> > > >> > > wrote: >> > > >> > > > If you want to do a seamless migration. I think a better way is to >> > build >> > > a >> > > > cross datacenter Kafka cluster temporarily. So the process is: >> > > > 1. Add several new Kafka brokers in your new datacenter and add >> them to >> > > > the old cluster. >> > > > 2. Use replica assignment tool to reassign all the partitions to >> > brokers >> > > > in new datacenter. >> > > > 3. Perform controlled shutdown on the brokers in old datacenter. >> > > > >> > > > Jiangjie (Becket) Qin >> > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" <kumarsharma.ni...@gmail.com> >> > wrote: >> > > > >> > > > >Hi Team, >> > > > > >> > > > >in my project, we have built a new datacenter for Kafka brokers and >> > > wants >> > > > >to migrate from current datacenter to new one. >> > > > > >> > > > >Switching producers and consumers wont be a problem provided New >> > > > >Datacenter >> > > > >has all the messages of existing Datacenter. >> > > > > >> > > > > >> > > > >i have only 1 topic with 2 partition that need to be migrated... >> that >> > > too >> > > > >it is only 1 time activity. >> > > > > >> > > > >Kindly suggest the best way to deal with this situation. >> > > > > >> > > > > >> > > > >Regards, >> > > > >Nitin Kumar Sharma. >> > > > >> > > > >> > > >> > > >> > > -- >> > > -Regards, >> > > Mayuresh R. Gharat >> > > (862) 250-7125 >> > > >> > >> >> >> >> -- >> Regards, >> Tao >> > > -- Regards, Tao
