hi, sorry for late response. ... i have been able to fix the issue .. problem was in my approach. I got confused between my source and target system while defining consumer & producer property file .. it is fixed now
Now new issue.. the rate at which data is migrated is very very slow... i mean it took 5 min to copy only 15Kb.. :( .. here are the property for producer and consumer.. there is no network latency between Source and Destination clusters as such. #### Producer ########### metadata.broker.list=<broker1IP>:9092,<broker2IP>:9092 serializer.class=kafka.serializer.DefaultEncoder auto.create.topics.enable=true request.required.acks=1 request.required.acks=1 producer.type=async batch.num.messages=3000 queue.buffering.max.ms=5000 queue.buffering.max.messages=100000 queue.enqueue.timeout.ms=-1 socket.send.buffer.bytes=5282880 ######### Consumer ########### zookeeper.connect=<zk1hostname>:2181,<zk2hostname>:2181,<zk3hostname>:2181 group.id=KafkaMaker auto.create.topics.enable=true socket.receive.buffer.bytes=5243880 zookeeper.connection.timeout.ms=1000000 num.consumer.fetchers=20 fetch.message.max.bytes=5243880 Regards, Nitin Kumar Sharma. On Tue, Mar 31, 2015 at 12:36 PM, tao xiao <xiaotao...@gmail.com> wrote: > Can you attach your mirror maker log? > > On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma <kumarsharma.ni...@gmail.com > > > wrote: > > > i tried with auto.offset.reset=smallest, but still not working.. > > > > there is data in my source cluster > > > > Regards, > > Nitin Kumar Sharma. > > > > > > On Mon, Mar 30, 2015 at 10:30 PM, tao xiao <xiaotao...@gmail.com> wrote: > > > > > Do you have data sending to *testtopic? *By default mirror maker only > > > consumes data being sent after it taps into the topic. you need to keep > > > sending data to the topic after mirror maker connection is established. > > If > > > you want to change the behavior you can set auto.offset.reset=smallest > so > > > that any new mirror maker coming to the topic will start from the > > smallest > > > offset > > > > > > On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma < > > kumarsharma.ni...@gmail.com > > > > > > > wrote: > > > > > > > thanks Xiao > > > > > > > > I tried MirrorMaker option in my test environment but failed. I am > not > > > > able to see the log getting copied to destination cluster. I see in > the > > > log > > > > of MirrorMaker process that connection is successfully established > > > between > > > > source and destination cluster but still not sure what i causing the > > > problem > > > > > > > > Env. Setup ==> > > > > > > > > I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) > > and > > > > 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03). > > > > Destination Clustern (Qenv05) -- i have 2 broker > > (Qenv05kf01,Qenv05kf02) > > > > and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03). > > > > > > > > II). i have kept consumer and producer properties file in one of the > > > > source kafka broker config folder. > > > > > > > > III).i have executed following command from the same kafka broker to > > > > start the process.. log are attached : > > > > > > > > /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker > > > -consumer.config > > > > /app/kafka/config/consumer1.properties --num.streams=2 > > --producer.config > > > > /app/kafka/config/producer1.properties --whitelist testtopic > > > > > > > > > > > > IV). I tried Consumer offset tracker tool also, while Mirror Maker > > > running > > > > . I tried by launching second session of same broker where mirror > maker > > > is > > > > running. I got error message that "*NoNode for > > > > /consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached. > > > > > > > > > > > > Regards, > > > > Nitin Kumar Sharma. > > > > > > > > > > > > On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xiaotao...@gmail.com> > > wrote: > > > > > > > >> Both consumer-1 and consumer-2 are properties of source clusters > > mirror > > > >> maker transfers data from. Mirror maker is designed to be able to > > > consume > > > >> data from N sources (N >= 1) and transfer data to one destination > > > cluster. > > > >> You are free to supply as many consumer properties as you want to > > > instruct > > > >> mirror maker where to consumer data from. > > > >> > > > >> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma < > > > >> kumarsharma.ni...@gmail.com> > > > >> wrote: > > > >> > > > >> > thanks Mayuresh and Jiangjie for your response. > > > >> > > > > >> > I have actually not understood Mirror maker clearly and hence bit > > > >> skeptical > > > >> > if i will be able to execute it effectively. > > > >> > > > > >> > Online i hv seen the following command to be execute, but not > > > understood > > > >> > what is consumer-1 & -2.properties here? do i need to copy from > my > > > >> > consumer code? also, any reason why to provide consumer property? > > > >> > > > > >> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config > > > >> > consumer-1.properties --consumer.config consumer-2.properties > > > >> > --producer.config producer.properties --whitelist my-topic > > > >> > > > > >> > > > > >> > Regards, > > > >> > Nitin Kumar Sharma. > > > >> > > > > >> > > > > >> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat < > > > >> > gharatmayures...@gmail.com > > > >> > > wrote: > > > >> > > > > >> > > You can use the Mirror maker to move data from one data center > to > > > >> other > > > >> > and > > > >> > > once all the data has been moved you can shut down the source > data > > > >> center > > > >> > > by doing a controlled shutdown. > > > >> > > > > > >> > > Thanks, > > > >> > > > > > >> > > Mayuresh > > > >> > > > > > >> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin > > > >> <j...@linkedin.com.invalid > > > >> > > > > > >> > > wrote: > > > >> > > > > > >> > > > If you want to do a seamless migration. I think a better way > is > > to > > > >> > build > > > >> > > a > > > >> > > > cross datacenter Kafka cluster temporarily. So the process is: > > > >> > > > 1. Add several new Kafka brokers in your new datacenter and > add > > > >> them to > > > >> > > > the old cluster. > > > >> > > > 2. Use replica assignment tool to reassign all the partitions > to > > > >> > brokers > > > >> > > > in new datacenter. > > > >> > > > 3. Perform controlled shutdown on the brokers in old > datacenter. > > > >> > > > > > > >> > > > Jiangjie (Becket) Qin > > > >> > > > > > > >> > > > On 3/25/15, 2:01 PM, "nitin sharma" < > > kumarsharma.ni...@gmail.com> > > > >> > wrote: > > > >> > > > > > > >> > > > >Hi Team, > > > >> > > > > > > > >> > > > >in my project, we have built a new datacenter for Kafka > brokers > > > and > > > >> > > wants > > > >> > > > >to migrate from current datacenter to new one. > > > >> > > > > > > > >> > > > >Switching producers and consumers wont be a problem provided > > New > > > >> > > > >Datacenter > > > >> > > > >has all the messages of existing Datacenter. > > > >> > > > > > > > >> > > > > > > > >> > > > >i have only 1 topic with 2 partition that need to be > > migrated... > > > >> that > > > >> > > too > > > >> > > > >it is only 1 time activity. > > > >> > > > > > > > >> > > > >Kindly suggest the best way to deal with this situation. > > > >> > > > > > > > >> > > > > > > > >> > > > >Regards, > > > >> > > > >Nitin Kumar Sharma. > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > -- > > > >> > > -Regards, > > > >> > > Mayuresh R. Gharat > > > >> > > (862) 250-7125 > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> Regards, > > > >> Tao > > > >> > > > > > > > > > > > > > > > > > -- > > > Regards, > > > Tao > > > > > > > > > -- > Regards, > Tao >
