thanks Xiao
I tried MirrorMaker option in my test environment but failed. I am not able
to see the log getting copied to destination cluster. I see in the log of
MirrorMaker process that connection is successfully established between
source and destination cluster but still not sure what i causing the problem
Env. Setup ==>
I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) and 3
zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
Destination Clustern (Qenv05) -- i have 2 broker (Qenv05kf01,Qenv05kf02)
and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
II). i have kept consumer and producer properties file in one of the source
kafka broker config folder.
III).i have executed following command from the same kafka broker to start
the process.. log are attached :
/app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker -consumer.config
/app/kafka/config/consumer1.properties --num.streams=2 --producer.config
/app/kafka/config/producer1.properties --whitelist testtopic
IV). I tried Consumer offset tracker tool also, while Mirror Maker running
. I tried by launching second session of same broker where mirror maker is
running. I got error message that "*NoNode for
/consumers/KafkaMaker/offsets/testtopic/0*" .Complete log attached.
Regards,
Nitin Kumar Sharma.
On Thu, Mar 26, 2015 at 11:24 AM, tao xiao <xiaotao...@gmail.com> wrote:
> Both consumer-1 and consumer-2 are properties of source clusters mirror
> maker transfers data from. Mirror maker is designed to be able to consume
> data from N sources (N >= 1) and transfer data to one destination cluster.
> You are free to supply as many consumer properties as you want to instruct
> mirror maker where to consumer data from.
>
> On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma <kumarsharma.ni...@gmail.com
> >
> wrote:
>
> > thanks Mayuresh and Jiangjie for your response.
> >
> > I have actually not understood Mirror maker clearly and hence bit
> skeptical
> > if i will be able to execute it effectively.
> >
> > Online i hv seen the following command to be execute, but not understood
> > what is consumer-1 & -2.properties here? do i need to copy from my
> > consumer code? also, any reason why to provide consumer property?
> >
> > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> > consumer-1.properties --consumer.config consumer-2.properties
> > --producer.config producer.properties --whitelist my-topic
> >
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > You can use the Mirror maker to move data from one data center to other
> > and
> > > once all the data has been moved you can shut down the source data
> center
> > > by doing a controlled shutdown.
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
> <j...@linkedin.com.invalid
> > >
> > > wrote:
> > >
> > > > If you want to do a seamless migration. I think a better way is to
> > build
> > > a
> > > > cross datacenter Kafka cluster temporarily. So the process is:
> > > > 1. Add several new Kafka brokers in your new datacenter and add them
> to
> > > > the old cluster.
> > > > 2. Use replica assignment tool to reassign all the partitions to
> > brokers
> > > > in new datacenter.
> > > > 3. Perform controlled shutdown on the brokers in old datacenter.
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > > On 3/25/15, 2:01 PM, "nitin sharma" <kumarsharma.ni...@gmail.com>
> > wrote:
> > > >
> > > > >Hi Team,
> > > > >
> > > > >in my project, we have built a new datacenter for Kafka brokers and
> > > wants
> > > > >to migrate from current datacenter to new one.
> > > > >
> > > > >Switching producers and consumers wont be a problem provided New
> > > > >Datacenter
> > > > >has all the messages of existing Datacenter.
> > > > >
> > > > >
> > > > >i have only 1 topic with 2 partition that need to be migrated...
> that
> > > too
> > > > >it is only 1 time activity.
> > > > >
> > > > >Kindly suggest the best way to deal with this situation.
> > > > >
> > > > >
> > > > >Regards,
> > > > >Nitin Kumar Sharma.
> > > >
> > > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>
>
>
> --
> Regards,
> Tao
>
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect
Qenv02zk01:2181 --topic testtopic --group KafkaMaker
Group Topic Pid Offset logSize
Lag Owner
Exception in thread "main" org.I0Itec.zkclient.exception.ZkNoNodeException:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
for /consumers/KafkaMaker/offsets/testtopic/0
at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
at kafka.utils.ZkUtils$.readData(ZkUtils.scala:461)
at
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply$mcVI$sp(ConsumerOffsetChecker.scala:89)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:89)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:89)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processTopic(ConsumerOffsetChecker.scala:88)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:153)
at
kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:153)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:76)
at
kafka.tools.ConsumerOffsetChecker$.main(ConsumerOffsetChecker.scala:152)
at kafka.tools.ConsumerOffsetChecker.main(ConsumerOffsetChecker.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /consumers/KafkaMaker/offsets/testtopic/0
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956)
at org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103)
at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770)
at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
... 16 more
