Re: Kafka server relocation

[tao xiao]

num.consumer.fetchers means the max number of fetcher threads that can be
spawned. it doesn't necessarily mean you can get as many fetcher threads as
you specify.

To me the metrics are suggesting a very slow consumption rate only 18.21
bytes/minute. Here is the benchmark Linkedin does

http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

You should check if 18.21 bytes/minute is the max throughput you can get on
your machine with bin/kafka-consumer-perf-test.sh if this is case you
definitely need to tune it

On Mon, Apr 13, 2015 at 12:43 PM, nitin sharma kumarsharma.ni...@gmail.com
wrote:

 hi Xiao,

 i have finally got JMX monitoring enabled for my kafka nodes in test
 envrionment and here is what i observed.
 i was monitoring mbeans under kafka.consumer domain of JVM running Kafka
 Mirror Maker process.

 =
 AllTopicsBytes === 18.21 bytes/minute
 FetchRequestRateAndTimeMs === 9.69 Request/min and  99th Percentile is
 104.13ms.
 ==

 Interesting thing is that I have specified num.consumer.fetchers=200 in
 my consumer property file but i can see only 8 threads of type :


 kafka.consumer:name=KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_zkhost-1428952277321-5e044226-138-1-host_brokerhostname-port_9092-FetchRequestRateAndTimeMs,type=FetchRequestAndResponseMetrics


 Could this be the issue?

 note, my JVM is set to 1GB and only 30MB is utilized most of the time.


 Regards,
 Nitin Kumar Sharma.


 On Wed, Apr 8, 2015 at 10:48 PM, tao xiao xiaotao...@gmail.com wrote:

  Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check
 if
  the consumer has problem processing messages
 
  On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma 
 kumarsharma.ni...@gmail.com
  wrote:
 
   thanks, but can you please tell which metrics could highlight the
 factor
   causing slow data migration by MirrorMaker?
  
   Regards,
   Nitin Kumar Sharma.
  
  
   On Tue, Apr 7, 2015 at 10:10 PM, tao xiao xiaotao...@gmail.com
 wrote:
  
You may need to look into the consumer metrics and producer metrics
 to
identify the root cause. metrics in kafka.consumer and kafka.producer
categories will help you find out the problems.
   
This link gives instruction how to read the metrics
http://kafka.apache.org/documentation.html#monitoring
   
   
On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma 
   kumarsharma.ni...@gmail.com
wrote:
   
 hi,

 sorry for late response. ... i have been able to fix the issue ..
   problem
 was in my approach. I got confused between my source and target
  system
 while defining consumer  producer property file .. it is fixed now

 Now new issue.. the rate at which data is migrated is very very
   slow... i
 mean it took 5 min to copy only 15Kb.. :( .. here are the property
  for
 producer and consumer.. there is no network latency between Source
  and
 Destination clusters as such.


  Producer ###
 metadata.broker.list=broker1IP:9092,broker2IP:9092
 serializer.class=kafka.serializer.DefaultEncoder
 auto.create.topics.enable=true
 request.required.acks=1
 request.required.acks=1
 producer.type=async
 batch.num.messages=3000
 queue.buffering.max.ms=5000
 queue.buffering.max.messages=10
 queue.enqueue.timeout.ms=-1
 socket.send.buffer.bytes=5282880

 # Consumer ###

   
  
 
 zookeeper.connect=zk1hostname:2181,zk2hostname:2181,zk3hostname:2181
 group.id=KafkaMaker
 auto.create.topics.enable=true
 socket.receive.buffer.bytes=5243880
 zookeeper.connection.timeout.ms=100
 num.consumer.fetchers=20
 fetch.message.max.bytes=5243880




 Regards,
 Nitin Kumar Sharma.


 On Tue, Mar 31, 2015 at 12:36 PM, tao xiao xiaotao...@gmail.com
   wrote:

  Can you attach your mirror maker log?
 
  On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma 
 kumarsharma.ni...@gmail.com
  
  wrote:
 
   i tried with auto.offset.reset=smallest, but still not
 working..
  
   there is data in my source cluster
  
   Regards,
   Nitin Kumar Sharma.
  
  
   On Mon, Mar 30, 2015 at 10:30 PM, tao xiao 
 xiaotao...@gmail.com
  
 wrote:
  
Do you have data sending to *testtopic? *By default mirror
  maker
only
consumes data being sent after it taps into the topic. you
 need
   to
 keep
sending data to the topic after mirror maker connection is
 established.
   If
you want to change the behavior you can set
 auto.offset.reset=smallest
  so
that any new mirror maker coming to the topic will start from
  the
   smallest
offset
   
On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma 
   kumarsharma.ni...@gmail.com

wrote:
   
 thanks 

Re: Kafka server relocation

[nitin sharma]

hi Xiao,

i have finally got JMX monitoring enabled for my kafka nodes in test
envrionment and here is what i observed.
i was monitoring mbeans under kafka.consumer domain of JVM running Kafka
Mirror Maker process.

=
AllTopicsBytes === 18.21 bytes/minute
FetchRequestRateAndTimeMs === 9.69 Request/min and  99th Percentile is
104.13ms.
==

Interesting thing is that I have specified num.consumer.fetchers=200 in
my consumer property file but i can see only 8 threads of type :

kafka.consumer:name=KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_zkhost-1428952277321-5e044226-138-1-host_brokerhostname-port_9092-FetchRequestRateAndTimeMs,type=FetchRequestAndResponseMetrics


Could this be the issue?

note, my JVM is set to 1GB and only 30MB is utilized most of the time.


Regards,
Nitin Kumar Sharma.


On Wed, Apr 8, 2015 at 10:48 PM, tao xiao xiaotao...@gmail.com wrote:

 Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check if
 the consumer has problem processing messages

 On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma kumarsharma.ni...@gmail.com
 wrote:

  thanks, but can you please tell which metrics could highlight the factor
  causing slow data migration by MirrorMaker?
 
  Regards,
  Nitin Kumar Sharma.
 
 
  On Tue, Apr 7, 2015 at 10:10 PM, tao xiao xiaotao...@gmail.com wrote:
 
   You may need to look into the consumer metrics and producer metrics to
   identify the root cause. metrics in kafka.consumer and kafka.producer
   categories will help you find out the problems.
  
   This link gives instruction how to read the metrics
   http://kafka.apache.org/documentation.html#monitoring
  
  
   On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma 
  kumarsharma.ni...@gmail.com
   wrote:
  
hi,
   
sorry for late response. ... i have been able to fix the issue ..
  problem
was in my approach. I got confused between my source and target
 system
while defining consumer  producer property file .. it is fixed now
   
Now new issue.. the rate at which data is migrated is very very
  slow... i
mean it took 5 min to copy only 15Kb.. :( .. here are the property
 for
producer and consumer.. there is no network latency between Source
 and
Destination clusters as such.
   
   
 Producer ###
metadata.broker.list=broker1IP:9092,broker2IP:9092
serializer.class=kafka.serializer.DefaultEncoder
auto.create.topics.enable=true
request.required.acks=1
request.required.acks=1
producer.type=async
batch.num.messages=3000
queue.buffering.max.ms=5000
queue.buffering.max.messages=10
queue.enqueue.timeout.ms=-1
socket.send.buffer.bytes=5282880
   
# Consumer ###
   
  
 
 zookeeper.connect=zk1hostname:2181,zk2hostname:2181,zk3hostname:2181
group.id=KafkaMaker
auto.create.topics.enable=true
socket.receive.buffer.bytes=5243880
zookeeper.connection.timeout.ms=100
num.consumer.fetchers=20
fetch.message.max.bytes=5243880
   
   
   
   
Regards,
Nitin Kumar Sharma.
   
   
On Tue, Mar 31, 2015 at 12:36 PM, tao xiao xiaotao...@gmail.com
  wrote:
   
 Can you attach your mirror maker log?

 On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma 
kumarsharma.ni...@gmail.com
 
 wrote:

  i tried with auto.offset.reset=smallest, but still not working..
 
  there is data in my source cluster
 
  Regards,
  Nitin Kumar Sharma.
 
 
  On Mon, Mar 30, 2015 at 10:30 PM, tao xiao xiaotao...@gmail.com
 
wrote:
 
   Do you have data sending to *testtopic? *By default mirror
 maker
   only
   consumes data being sent after it taps into the topic. you need
  to
keep
   sending data to the topic after mirror maker connection is
established.
  If
   you want to change the behavior you can set
auto.offset.reset=smallest
 so
   that any new mirror maker coming to the topic will start from
 the
  smallest
   offset
  
   On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma 
  kumarsharma.ni...@gmail.com
   
   wrote:
  
thanks Xiao
   
I tried MirrorMaker option in my test environment but
 failed. I
   am
 not
able to see the log getting copied to destination cluster. I
  see
   in
 the
   log
of MirrorMaker process that connection is successfully
   established
   between
source and destination cluster but still not sure what i
  causing
the
   problem
   
Env. Setup ==
   
I). Source Cluster (Qenv02) -- i have 2
broker(Qenv02kf01,Qenv02kf02)
  and
3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
Destination Clustern (Qenv05) -- i have 2 broker
  (Qenv05kf01,Qenv05kf02)
and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
   
II). i have kept consumer and producer properties file in one
  of
the

Re: Kafka server relocation

[tao xiao]

how about the consumer lag of mirror maker?

On Mon, Apr 13, 2015 at 1:33 PM, nitin sharma kumarsharma.ni...@gmail.com
wrote:

 i just tested that too and below is the stats.. it is clear that with
 kafka-consumer-perf-test.sh, i am able to get a high throughput. around
 44.0213 MB/sec.

 Seriously some configuration needs to be tweaked in MirrorMaker
 configuration for speedy processing... Can you think of something ?


 start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec,
 data.consumed.in.nMsg, nMsg.sec
 2015-04-13 20:29:29:070, 2015-04-13 20:29:34:596, 1048576, 23.1552,
 44.0213, 16683, 31716.7300

 Regards,
 Nitin Kumar Sharma.


 On Mon, Apr 13, 2015 at 3:51 PM, tao xiao xiaotao...@gmail.com wrote:

  num.consumer.fetchers means the max number of fetcher threads that can be
  spawned. it doesn't necessarily mean you can get as many fetcher threads
 as
  you specify.
 
  To me the metrics are suggesting a very slow consumption rate only 18.21
  bytes/minute. Here is the benchmark Linkedin does
 
 
 
 http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
 
  You should check if 18.21 bytes/minute is the max throughput you can get
 on
  your machine with bin/kafka-consumer-perf-test.sh if this is case you
  definitely need to tune it
 
  On Mon, Apr 13, 2015 at 12:43 PM, nitin sharma 
  kumarsharma.ni...@gmail.com
  wrote:
 
   hi Xiao,
  
   i have finally got JMX monitoring enabled for my kafka nodes in test
   envrionment and here is what i observed.
   i was monitoring mbeans under kafka.consumer domain of JVM running
  Kafka
   Mirror Maker process.
  
   =
   AllTopicsBytes === 18.21 bytes/minute
   FetchRequestRateAndTimeMs === 9.69 Request/min and  99th Percentile is
   104.13ms.
   ==
  
   Interesting thing is that I have specified num.consumer.fetchers=200
 in
   my consumer property file but i can see only 8 threads of type :
  
  
  
 
 kafka.consumer:name=KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_zkhost-1428952277321-5e044226-138-1-host_brokerhostname-port_9092-FetchRequestRateAndTimeMs,type=FetchRequestAndResponseMetrics
  
  
   Could this be the issue?
  
   note, my JVM is set to 1GB and only 30MB is utilized most of the time.
  
  
   Regards,
   Nitin Kumar Sharma.
  
  
   On Wed, Apr 8, 2015 at 10:48 PM, tao xiao xiaotao...@gmail.com
 wrote:
  
Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to
  check
   if
the consumer has problem processing messages
   
On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma 
   kumarsharma.ni...@gmail.com
wrote:
   
 thanks, but can you please tell which metrics could highlight the
   factor
 causing slow data migration by MirrorMaker?

 Regards,
 Nitin Kumar Sharma.


 On Tue, Apr 7, 2015 at 10:10 PM, tao xiao xiaotao...@gmail.com
   wrote:

  You may need to look into the consumer metrics and producer
 metrics
   to
  identify the root cause. metrics in kafka.consumer and
  kafka.producer
  categories will help you find out the problems.
 
  This link gives instruction how to read the metrics
  http://kafka.apache.org/documentation.html#monitoring
 
 
  On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma 
 kumarsharma.ni...@gmail.com
  wrote:
 
   hi,
  
   sorry for late response. ... i have been able to fix the issue
 ..
 problem
   was in my approach. I got confused between my source and target
system
   while defining consumer  producer property file .. it is fixed
  now
  
   Now new issue.. the rate at which data is migrated is very very
 slow... i
   mean it took 5 min to copy only 15Kb.. :( .. here are the
  property
for
   producer and consumer.. there is no network latency between
  Source
and
   Destination clusters as such.
  
  
    Producer ###
   metadata.broker.list=broker1IP:9092,broker2IP:9092
   serializer.class=kafka.serializer.DefaultEncoder
   auto.create.topics.enable=true
   request.required.acks=1
   request.required.acks=1
   producer.type=async
   batch.num.messages=3000
   queue.buffering.max.ms=5000
   queue.buffering.max.messages=10
   queue.enqueue.timeout.ms=-1
   socket.send.buffer.bytes=5282880
  
   # Consumer ###
  
 

   
  
 
 zookeeper.connect=zk1hostname:2181,zk2hostname:2181,zk3hostname:2181
   group.id=KafkaMaker
   auto.create.topics.enable=true
   socket.receive.buffer.bytes=5243880
   zookeeper.connection.timeout.ms=100
   num.consumer.fetchers=20
   fetch.message.max.bytes=5243880
  
  
  
  
   Regards,
   Nitin Kumar Sharma.
  
  
   On Tue, Mar 31, 2015 at 12:36 PM, tao xiao 
 xiaotao...@gmail.com
  
 wrote:
  
Can you attach your mirror maker 

Re: Kafka server relocation

[nitin sharma]

i just tested that too and below is the stats.. it is clear that with
kafka-consumer-perf-test.sh, i am able to get a high throughput. around
44.0213 MB/sec.

Seriously some configuration needs to be tweaked in MirrorMaker
configuration for speedy processing... Can you think of something ?


start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec,
data.consumed.in.nMsg, nMsg.sec
2015-04-13 20:29:29:070, 2015-04-13 20:29:34:596, 1048576, 23.1552,
44.0213, 16683, 31716.7300

Regards,
Nitin Kumar Sharma.


On Mon, Apr 13, 2015 at 3:51 PM, tao xiao xiaotao...@gmail.com wrote:

 num.consumer.fetchers means the max number of fetcher threads that can be
 spawned. it doesn't necessarily mean you can get as many fetcher threads as
 you specify.

 To me the metrics are suggesting a very slow consumption rate only 18.21
 bytes/minute. Here is the benchmark Linkedin does


 http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

 You should check if 18.21 bytes/minute is the max throughput you can get on
 your machine with bin/kafka-consumer-perf-test.sh if this is case you
 definitely need to tune it

 On Mon, Apr 13, 2015 at 12:43 PM, nitin sharma 
 kumarsharma.ni...@gmail.com
 wrote:

  hi Xiao,
 
  i have finally got JMX monitoring enabled for my kafka nodes in test
  envrionment and here is what i observed.
  i was monitoring mbeans under kafka.consumer domain of JVM running
 Kafka
  Mirror Maker process.
 
  =
  AllTopicsBytes === 18.21 bytes/minute
  FetchRequestRateAndTimeMs === 9.69 Request/min and  99th Percentile is
  104.13ms.
  ==
 
  Interesting thing is that I have specified num.consumer.fetchers=200 in
  my consumer property file but i can see only 8 threads of type :
 
 
 
 kafka.consumer:name=KafkaMaker1-ConsumerFetcherThread-KafkaMaker1_zkhost-1428952277321-5e044226-138-1-host_brokerhostname-port_9092-FetchRequestRateAndTimeMs,type=FetchRequestAndResponseMetrics
 
 
  Could this be the issue?
 
  note, my JVM is set to 1GB and only 30MB is utilized most of the time.
 
 
  Regards,
  Nitin Kumar Sharma.
 
 
  On Wed, Apr 8, 2015 at 10:48 PM, tao xiao xiaotao...@gmail.com wrote:
 
   Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to
 check
  if
   the consumer has problem processing messages
  
   On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma 
  kumarsharma.ni...@gmail.com
   wrote:
  
thanks, but can you please tell which metrics could highlight the
  factor
causing slow data migration by MirrorMaker?
   
Regards,
Nitin Kumar Sharma.
   
   
On Tue, Apr 7, 2015 at 10:10 PM, tao xiao xiaotao...@gmail.com
  wrote:
   
 You may need to look into the consumer metrics and producer metrics
  to
 identify the root cause. metrics in kafka.consumer and
 kafka.producer
 categories will help you find out the problems.

 This link gives instruction how to read the metrics
 http://kafka.apache.org/documentation.html#monitoring


 On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma 
kumarsharma.ni...@gmail.com
 wrote:

  hi,
 
  sorry for late response. ... i have been able to fix the issue ..
problem
  was in my approach. I got confused between my source and target
   system
  while defining consumer  producer property file .. it is fixed
 now
 
  Now new issue.. the rate at which data is migrated is very very
slow... i
  mean it took 5 min to copy only 15Kb.. :( .. here are the
 property
   for
  producer and consumer.. there is no network latency between
 Source
   and
  Destination clusters as such.
 
 
   Producer ###
  metadata.broker.list=broker1IP:9092,broker2IP:9092
  serializer.class=kafka.serializer.DefaultEncoder
  auto.create.topics.enable=true
  request.required.acks=1
  request.required.acks=1
  producer.type=async
  batch.num.messages=3000
  queue.buffering.max.ms=5000
  queue.buffering.max.messages=10
  queue.enqueue.timeout.ms=-1
  socket.send.buffer.bytes=5282880
 
  # Consumer ###
 

   
  
 
 zookeeper.connect=zk1hostname:2181,zk2hostname:2181,zk3hostname:2181
  group.id=KafkaMaker
  auto.create.topics.enable=true
  socket.receive.buffer.bytes=5243880
  zookeeper.connection.timeout.ms=100
  num.consumer.fetchers=20
  fetch.message.max.bytes=5243880
 
 
 
 
  Regards,
  Nitin Kumar Sharma.
 
 
  On Tue, Mar 31, 2015 at 12:36 PM, tao xiao xiaotao...@gmail.com
 
wrote:
 
   Can you attach your mirror maker log?
  
   On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma 
  kumarsharma.ni...@gmail.com
   
   wrote:
  
i tried with auto.offset.reset=smallest, but still not
  working..
   
there is data in my source cluster
   
Regards,
Nitin 

Re: Kafka server relocation

[nitin sharma]

thanks, but can you please tell which metrics could highlight the factor
causing slow data migration by MirrorMaker?

Regards,
Nitin Kumar Sharma.


On Tue, Apr 7, 2015 at 10:10 PM, tao xiao xiaotao...@gmail.com wrote:

 You may need to look into the consumer metrics and producer metrics to
 identify the root cause. metrics in kafka.consumer and kafka.producer
 categories will help you find out the problems.

 This link gives instruction how to read the metrics
 http://kafka.apache.org/documentation.html#monitoring


 On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma kumarsharma.ni...@gmail.com
 wrote:

  hi,
 
  sorry for late response. ... i have been able to fix the issue .. problem
  was in my approach. I got confused between my source and target system
  while defining consumer  producer property file .. it is fixed now
 
  Now new issue.. the rate at which data is migrated is very very slow... i
  mean it took 5 min to copy only 15Kb.. :( .. here are the property for
  producer and consumer.. there is no network latency between Source and
  Destination clusters as such.
 
 
   Producer ###
  metadata.broker.list=broker1IP:9092,broker2IP:9092
  serializer.class=kafka.serializer.DefaultEncoder
  auto.create.topics.enable=true
  request.required.acks=1
  request.required.acks=1
  producer.type=async
  batch.num.messages=3000
  queue.buffering.max.ms=5000
  queue.buffering.max.messages=10
  queue.enqueue.timeout.ms=-1
  socket.send.buffer.bytes=5282880
 
  # Consumer ###
 
 zookeeper.connect=zk1hostname:2181,zk2hostname:2181,zk3hostname:2181
  group.id=KafkaMaker
  auto.create.topics.enable=true
  socket.receive.buffer.bytes=5243880
  zookeeper.connection.timeout.ms=100
  num.consumer.fetchers=20
  fetch.message.max.bytes=5243880
 
 
 
 
  Regards,
  Nitin Kumar Sharma.
 
 
  On Tue, Mar 31, 2015 at 12:36 PM, tao xiao xiaotao...@gmail.com wrote:
 
   Can you attach your mirror maker log?
  
   On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma 
  kumarsharma.ni...@gmail.com
   
   wrote:
  
i tried with auto.offset.reset=smallest, but still not working..
   
there is data in my source cluster
   
Regards,
Nitin Kumar Sharma.
   
   
On Mon, Mar 30, 2015 at 10:30 PM, tao xiao xiaotao...@gmail.com
  wrote:
   
 Do you have data sending to *testtopic? *By default mirror maker
 only
 consumes data being sent after it taps into the topic. you need to
  keep
 sending data to the topic after mirror maker connection is
  established.
If
 you want to change the behavior you can set
  auto.offset.reset=smallest
   so
 that any new mirror maker coming to the topic will start from the
smallest
 offset

 On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma 
kumarsharma.ni...@gmail.com
 
 wrote:

  thanks Xiao
 
  I tried MirrorMaker option in my test environment but failed. I
 am
   not
  able to see the log getting copied to destination cluster. I see
 in
   the
 log
  of MirrorMaker process that connection is successfully
 established
 between
  source and destination cluster but still not sure what i causing
  the
 problem
 
  Env. Setup ==
 
  I). Source Cluster (Qenv02) -- i have 2
  broker(Qenv02kf01,Qenv02kf02)
and
  3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
  Destination Clustern (Qenv05) -- i have 2 broker
(Qenv05kf01,Qenv05kf02)
  and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
 
  II). i have kept consumer and producer properties file in one of
  the
  source kafka broker config folder.
 
  III).i have executed following command from the same kafka broker
  to
  start the process.. log are attached :
 
  /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
 -consumer.config
  /app/kafka/config/consumer1.properties --num.streams=2
--producer.config
  /app/kafka/config/producer1.properties --whitelist testtopic
 
 
  IV). I tried Consumer offset tracker tool also, while Mirror
 Maker
 running
  . I tried by launching second session of same broker where mirror
   maker
 is
  running. I got error message that *NoNode for
  /consumers/KafkaMaker/offsets/testtopic/0* .Complete log
 attached.
 
 
  Regards,
  Nitin Kumar Sharma.
 
 
  On Thu, Mar 26, 2015 at 11:24 AM, tao xiao xiaotao...@gmail.com
 
wrote:
 
  Both consumer-1 and consumer-2 are properties of source clusters
mirror
  maker transfers data from. Mirror maker is designed to be able
 to
 consume
  data from N sources (N = 1) and transfer data to one
 destination
 cluster.
  You are free to supply as many consumer properties as you want
 to
 instruct
  mirror maker where to consumer data from.
 
  On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma 
  kumarsharma.ni...@gmail.com
  wrote:
 
   thanks Mayuresh and 

Re: Kafka server relocation

[tao xiao]

Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check if
the consumer has problem processing messages

On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma kumarsharma.ni...@gmail.com
wrote:

 thanks, but can you please tell which metrics could highlight the factor
 causing slow data migration by MirrorMaker?

 Regards,
 Nitin Kumar Sharma.


 On Tue, Apr 7, 2015 at 10:10 PM, tao xiao xiaotao...@gmail.com wrote:

  You may need to look into the consumer metrics and producer metrics to
  identify the root cause. metrics in kafka.consumer and kafka.producer
  categories will help you find out the problems.
 
  This link gives instruction how to read the metrics
  http://kafka.apache.org/documentation.html#monitoring
 
 
  On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma 
 kumarsharma.ni...@gmail.com
  wrote:
 
   hi,
  
   sorry for late response. ... i have been able to fix the issue ..
 problem
   was in my approach. I got confused between my source and target system
   while defining consumer  producer property file .. it is fixed now
  
   Now new issue.. the rate at which data is migrated is very very
 slow... i
   mean it took 5 min to copy only 15Kb.. :( .. here are the property for
   producer and consumer.. there is no network latency between Source and
   Destination clusters as such.
  
  
    Producer ###
   metadata.broker.list=broker1IP:9092,broker2IP:9092
   serializer.class=kafka.serializer.DefaultEncoder
   auto.create.topics.enable=true
   request.required.acks=1
   request.required.acks=1
   producer.type=async
   batch.num.messages=3000
   queue.buffering.max.ms=5000
   queue.buffering.max.messages=10
   queue.enqueue.timeout.ms=-1
   socket.send.buffer.bytes=5282880
  
   # Consumer ###
  
 
 zookeeper.connect=zk1hostname:2181,zk2hostname:2181,zk3hostname:2181
   group.id=KafkaMaker
   auto.create.topics.enable=true
   socket.receive.buffer.bytes=5243880
   zookeeper.connection.timeout.ms=100
   num.consumer.fetchers=20
   fetch.message.max.bytes=5243880
  
  
  
  
   Regards,
   Nitin Kumar Sharma.
  
  
   On Tue, Mar 31, 2015 at 12:36 PM, tao xiao xiaotao...@gmail.com
 wrote:
  
Can you attach your mirror maker log?
   
On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma 
   kumarsharma.ni...@gmail.com

wrote:
   
 i tried with auto.offset.reset=smallest, but still not working..

 there is data in my source cluster

 Regards,
 Nitin Kumar Sharma.


 On Mon, Mar 30, 2015 at 10:30 PM, tao xiao xiaotao...@gmail.com
   wrote:

  Do you have data sending to *testtopic? *By default mirror maker
  only
  consumes data being sent after it taps into the topic. you need
 to
   keep
  sending data to the topic after mirror maker connection is
   established.
 If
  you want to change the behavior you can set
   auto.offset.reset=smallest
so
  that any new mirror maker coming to the topic will start from the
 smallest
  offset
 
  On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma 
 kumarsharma.ni...@gmail.com
  
  wrote:
 
   thanks Xiao
  
   I tried MirrorMaker option in my test environment but failed. I
  am
not
   able to see the log getting copied to destination cluster. I
 see
  in
the
  log
   of MirrorMaker process that connection is successfully
  established
  between
   source and destination cluster but still not sure what i
 causing
   the
  problem
  
   Env. Setup ==
  
   I). Source Cluster (Qenv02) -- i have 2
   broker(Qenv02kf01,Qenv02kf02)
 and
   3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
   Destination Clustern (Qenv05) -- i have 2 broker
 (Qenv05kf01,Qenv05kf02)
   and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
  
   II). i have kept consumer and producer properties file in one
 of
   the
   source kafka broker config folder.
  
   III).i have executed following command from the same kafka
 broker
   to
   start the process.. log are attached :
  
   /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
  -consumer.config
   /app/kafka/config/consumer1.properties --num.streams=2
 --producer.config
   /app/kafka/config/producer1.properties --whitelist testtopic
  
  
   IV). I tried Consumer offset tracker tool also, while Mirror
  Maker
  running
   . I tried by launching second session of same broker where
 mirror
maker
  is
   running. I got error message that *NoNode for
   /consumers/KafkaMaker/offsets/testtopic/0* .Complete log
  attached.
  
  
   Regards,
   Nitin Kumar Sharma.
  
  
   On Thu, Mar 26, 2015 at 11:24 AM, tao xiao 
 xiaotao...@gmail.com
  
 wrote:
  
   Both consumer-1 and consumer-2 are properties of source
 clusters
 mirror
   maker transfers data from. Mirror maker is designed to be able
 

Re: Kafka server relocation

[nitin sharma]

hi,

sorry for late response. ... i have been able to fix the issue .. problem
was in my approach. I got confused between my source and target system
while defining consumer  producer property file .. it is fixed now

Now new issue.. the rate at which data is migrated is very very slow... i
mean it took 5 min to copy only 15Kb.. :( .. here are the property for
producer and consumer.. there is no network latency between Source and
Destination clusters as such.


 Producer ###
metadata.broker.list=broker1IP:9092,broker2IP:9092
serializer.class=kafka.serializer.DefaultEncoder
auto.create.topics.enable=true
request.required.acks=1
request.required.acks=1
producer.type=async
batch.num.messages=3000
queue.buffering.max.ms=5000
queue.buffering.max.messages=10
queue.enqueue.timeout.ms=-1
socket.send.buffer.bytes=5282880

# Consumer ###
zookeeper.connect=zk1hostname:2181,zk2hostname:2181,zk3hostname:2181
group.id=KafkaMaker
auto.create.topics.enable=true
socket.receive.buffer.bytes=5243880
zookeeper.connection.timeout.ms=100
num.consumer.fetchers=20
fetch.message.max.bytes=5243880




Regards,
Nitin Kumar Sharma.


On Tue, Mar 31, 2015 at 12:36 PM, tao xiao xiaotao...@gmail.com wrote:

 Can you attach your mirror maker log?

 On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma kumarsharma.ni...@gmail.com
 
 wrote:

  i tried with auto.offset.reset=smallest, but still not working..
 
  there is data in my source cluster
 
  Regards,
  Nitin Kumar Sharma.
 
 
  On Mon, Mar 30, 2015 at 10:30 PM, tao xiao xiaotao...@gmail.com wrote:
 
   Do you have data sending to *testtopic? *By default mirror maker only
   consumes data being sent after it taps into the topic. you need to keep
   sending data to the topic after mirror maker connection is established.
  If
   you want to change the behavior you can set auto.offset.reset=smallest
 so
   that any new mirror maker coming to the topic will start from the
  smallest
   offset
  
   On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma 
  kumarsharma.ni...@gmail.com
   
   wrote:
  
thanks Xiao
   
I tried MirrorMaker option in my test environment but failed. I am
 not
able to see the log getting copied to destination cluster. I see in
 the
   log
of MirrorMaker process that connection is successfully established
   between
source and destination cluster but still not sure what i causing the
   problem
   
Env. Setup ==
   
I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02)
  and
3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
Destination Clustern (Qenv05) -- i have 2 broker
  (Qenv05kf01,Qenv05kf02)
and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
   
II). i have kept consumer and producer properties file in one of the
source kafka broker config folder.
   
III).i have executed following command from the same kafka broker  to
start the process.. log are attached :
   
/app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
   -consumer.config
/app/kafka/config/consumer1.properties --num.streams=2
  --producer.config
/app/kafka/config/producer1.properties --whitelist testtopic
   
   
IV). I tried Consumer offset tracker tool also, while Mirror Maker
   running
. I tried by launching second session of same broker where mirror
 maker
   is
running. I got error message that *NoNode for
/consumers/KafkaMaker/offsets/testtopic/0* .Complete log attached.
   
   
Regards,
Nitin Kumar Sharma.
   
   
On Thu, Mar 26, 2015 at 11:24 AM, tao xiao xiaotao...@gmail.com
  wrote:
   
Both consumer-1 and consumer-2 are properties of source clusters
  mirror
maker transfers data from. Mirror maker is designed to be able to
   consume
data from N sources (N = 1) and transfer data to one destination
   cluster.
You are free to supply as many consumer properties as you want to
   instruct
mirror maker where to consumer data from.
   
On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma 
kumarsharma.ni...@gmail.com
wrote:
   
 thanks Mayuresh and Jiangjie for your response.

 I have actually not understood Mirror maker clearly and hence bit
skeptical
 if i will be able to execute it effectively.

 Online i hv seen the following command to be execute, but not
   understood
  what is consumer-1  -2.properties here? do i need to copy from
 my
 consumer code? also, any reason why to provide consumer property?

 bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
 consumer-1.properties --consumer.config consumer-2.properties
 --producer.config producer.properties --whitelist my-topic


 Regards,
 Nitin Kumar Sharma.


 On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat 
 gharatmayures...@gmail.com
  wrote:

  You can use the Mirror maker to move data from one data center
 to
other
 and
  once all the data has been moved you 

Re: Kafka server relocation

[tao xiao]

You may need to look into the consumer metrics and producer metrics to
identify the root cause. metrics in kafka.consumer and kafka.producer
categories will help you find out the problems.

This link gives instruction how to read the metrics
http://kafka.apache.org/documentation.html#monitoring


On Wed, Apr 8, 2015 at 3:39 AM, nitin sharma kumarsharma.ni...@gmail.com
wrote:

 hi,

 sorry for late response. ... i have been able to fix the issue .. problem
 was in my approach. I got confused between my source and target system
 while defining consumer  producer property file .. it is fixed now

 Now new issue.. the rate at which data is migrated is very very slow... i
 mean it took 5 min to copy only 15Kb.. :( .. here are the property for
 producer and consumer.. there is no network latency between Source and
 Destination clusters as such.


  Producer ###
 metadata.broker.list=broker1IP:9092,broker2IP:9092
 serializer.class=kafka.serializer.DefaultEncoder
 auto.create.topics.enable=true
 request.required.acks=1
 request.required.acks=1
 producer.type=async
 batch.num.messages=3000
 queue.buffering.max.ms=5000
 queue.buffering.max.messages=10
 queue.enqueue.timeout.ms=-1
 socket.send.buffer.bytes=5282880

 # Consumer ###
 zookeeper.connect=zk1hostname:2181,zk2hostname:2181,zk3hostname:2181
 group.id=KafkaMaker
 auto.create.topics.enable=true
 socket.receive.buffer.bytes=5243880
 zookeeper.connection.timeout.ms=100
 num.consumer.fetchers=20
 fetch.message.max.bytes=5243880




 Regards,
 Nitin Kumar Sharma.


 On Tue, Mar 31, 2015 at 12:36 PM, tao xiao xiaotao...@gmail.com wrote:

  Can you attach your mirror maker log?
 
  On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma 
 kumarsharma.ni...@gmail.com
  
  wrote:
 
   i tried with auto.offset.reset=smallest, but still not working..
  
   there is data in my source cluster
  
   Regards,
   Nitin Kumar Sharma.
  
  
   On Mon, Mar 30, 2015 at 10:30 PM, tao xiao xiaotao...@gmail.com
 wrote:
  
Do you have data sending to *testtopic? *By default mirror maker only
consumes data being sent after it taps into the topic. you need to
 keep
sending data to the topic after mirror maker connection is
 established.
   If
you want to change the behavior you can set
 auto.offset.reset=smallest
  so
that any new mirror maker coming to the topic will start from the
   smallest
offset
   
On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma 
   kumarsharma.ni...@gmail.com

wrote:
   
 thanks Xiao

 I tried MirrorMaker option in my test environment but failed. I am
  not
 able to see the log getting copied to destination cluster. I see in
  the
log
 of MirrorMaker process that connection is successfully established
between
 source and destination cluster but still not sure what i causing
 the
problem

 Env. Setup ==

 I). Source Cluster (Qenv02) -- i have 2
 broker(Qenv02kf01,Qenv02kf02)
   and
 3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
 Destination Clustern (Qenv05) -- i have 2 broker
   (Qenv05kf01,Qenv05kf02)
 and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).

 II). i have kept consumer and producer properties file in one of
 the
 source kafka broker config folder.

 III).i have executed following command from the same kafka broker
 to
 start the process.. log are attached :

 /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
-consumer.config
 /app/kafka/config/consumer1.properties --num.streams=2
   --producer.config
 /app/kafka/config/producer1.properties --whitelist testtopic


 IV). I tried Consumer offset tracker tool also, while Mirror Maker
running
 . I tried by launching second session of same broker where mirror
  maker
is
 running. I got error message that *NoNode for
 /consumers/KafkaMaker/offsets/testtopic/0* .Complete log attached.


 Regards,
 Nitin Kumar Sharma.


 On Thu, Mar 26, 2015 at 11:24 AM, tao xiao xiaotao...@gmail.com
   wrote:

 Both consumer-1 and consumer-2 are properties of source clusters
   mirror
 maker transfers data from. Mirror maker is designed to be able to
consume
 data from N sources (N = 1) and transfer data to one destination
cluster.
 You are free to supply as many consumer properties as you want to
instruct
 mirror maker where to consumer data from.

 On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma 
 kumarsharma.ni...@gmail.com
 wrote:

  thanks Mayuresh and Jiangjie for your response.
 
  I have actually not understood Mirror maker clearly and hence
 bit
 skeptical
  if i will be able to execute it effectively.
 
  Online i hv seen the following command to be execute, but not
understood
   what is consumer-1  -2.properties here? do i need to copy from
  my
  consumer code? also, any reason why to 

Re: Kafka server relocation

[tao xiao]

Can you attach your mirror maker log?

On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma kumarsharma.ni...@gmail.com
wrote:

 i tried with auto.offset.reset=smallest, but still not working..

 there is data in my source cluster

 Regards,
 Nitin Kumar Sharma.


 On Mon, Mar 30, 2015 at 10:30 PM, tao xiao xiaotao...@gmail.com wrote:

  Do you have data sending to *testtopic? *By default mirror maker only
  consumes data being sent after it taps into the topic. you need to keep
  sending data to the topic after mirror maker connection is established.
 If
  you want to change the behavior you can set auto.offset.reset=smallest so
  that any new mirror maker coming to the topic will start from the
 smallest
  offset
 
  On Tue, Mar 31, 2015 at 3:53 AM, nitin sharma 
 kumarsharma.ni...@gmail.com
  
  wrote:
 
   thanks Xiao
  
   I tried MirrorMaker option in my test environment but failed. I am not
   able to see the log getting copied to destination cluster. I see in the
  log
   of MirrorMaker process that connection is successfully established
  between
   source and destination cluster but still not sure what i causing the
  problem
  
   Env. Setup ==
  
   I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02)
 and
   3 zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
   Destination Clustern (Qenv05) -- i have 2 broker
 (Qenv05kf01,Qenv05kf02)
   and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).
  
   II). i have kept consumer and producer properties file in one of the
   source kafka broker config folder.
  
   III).i have executed following command from the same kafka broker  to
   start the process.. log are attached :
  
   /app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker
  -consumer.config
   /app/kafka/config/consumer1.properties --num.streams=2
 --producer.config
   /app/kafka/config/producer1.properties --whitelist testtopic
  
  
   IV). I tried Consumer offset tracker tool also, while Mirror Maker
  running
   . I tried by launching second session of same broker where mirror maker
  is
   running. I got error message that *NoNode for
   /consumers/KafkaMaker/offsets/testtopic/0* .Complete log attached.
  
  
   Regards,
   Nitin Kumar Sharma.
  
  
   On Thu, Mar 26, 2015 at 11:24 AM, tao xiao xiaotao...@gmail.com
 wrote:
  
   Both consumer-1 and consumer-2 are properties of source clusters
 mirror
   maker transfers data from. Mirror maker is designed to be able to
  consume
   data from N sources (N = 1) and transfer data to one destination
  cluster.
   You are free to supply as many consumer properties as you want to
  instruct
   mirror maker where to consumer data from.
  
   On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma 
   kumarsharma.ni...@gmail.com
   wrote:
  
thanks Mayuresh and Jiangjie for your response.
   
I have actually not understood Mirror maker clearly and hence bit
   skeptical
if i will be able to execute it effectively.
   
Online i hv seen the following command to be execute, but not
  understood
 what is consumer-1  -2.properties here? do i need to copy from my
consumer code? also, any reason why to provide consumer property?
   
bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
consumer-1.properties --consumer.config consumer-2.properties
--producer.config producer.properties --whitelist my-topic
   
   
Regards,
Nitin Kumar Sharma.
   
   
On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat 
gharatmayures...@gmail.com
 wrote:
   
 You can use the Mirror maker to move data from one data center to
   other
and
 once all the data has been moved you can shut down the source data
   center
 by doing a controlled shutdown.

 Thanks,

 Mayuresh

 On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
   j...@linkedin.com.invalid

 wrote:

  If you want to do a seamless migration. I think a better way is
 to
build
 a
  cross datacenter Kafka cluster temporarily. So the process is:
  1. Add several new Kafka brokers in your new datacenter and add
   them to
  the old cluster.
  2. Use replica assignment tool to reassign all the partitions to
brokers
  in new datacenter.
  3. Perform controlled shutdown on the brokers in old datacenter.
 
  Jiangjie (Becket) Qin
 
  On 3/25/15, 2:01 PM, nitin sharma 
 kumarsharma.ni...@gmail.com
wrote:
 
  Hi Team,
  
  in my project, we have built a new datacenter for Kafka brokers
  and
 wants
  to migrate from current datacenter to new one.
  
  Switching producers and consumers wont be a problem provided
 New
  Datacenter
  has all the messages of existing Datacenter.
  
  
  i have only 1 topic with 2 partition that need to be
 migrated...
   that
 too
  it is only 1 time activity.
  
  Kindly suggest the best way to deal with this situation.
  
  
  Regards,
  Nitin Kumar 

Re: Kafka server relocation

[nitin sharma]

thanks Xiao

I tried MirrorMaker option in my test environment but failed. I am not able
to see the log getting copied to destination cluster. I see in the log of
MirrorMaker process that connection is successfully established between
source and destination cluster but still not sure what i causing the problem

Env. Setup ==

I). Source Cluster (Qenv02) -- i have 2 broker(Qenv02kf01,Qenv02kf02) and 3
zk(Qenv02zk01,Qenv02zk02 and Qenv02zk03).
Destination Clustern (Qenv05) -- i have 2 broker (Qenv05kf01,Qenv05kf02)
and 3 zk(Qenv05zk01,Qenv05zk02 and Qenv05zk03).

II). i have kept consumer and producer properties file in one of the source
kafka broker config folder.

III).i have executed following command from the same kafka broker  to start
the process.. log are attached :

/app/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker -consumer.config
/app/kafka/config/consumer1.properties --num.streams=2 --producer.config
/app/kafka/config/producer1.properties --whitelist testtopic


IV). I tried Consumer offset tracker tool also, while Mirror Maker running
. I tried by launching second session of same broker where mirror maker is
running. I got error message that *NoNode for
/consumers/KafkaMaker/offsets/testtopic/0* .Complete log attached.


Regards,
Nitin Kumar Sharma.


On Thu, Mar 26, 2015 at 11:24 AM, tao xiao xiaotao...@gmail.com wrote:

 Both consumer-1 and consumer-2 are properties of source clusters mirror
 maker transfers data from. Mirror maker is designed to be able to consume
 data from N sources (N = 1) and transfer data to one destination cluster.
 You are free to supply as many consumer properties as you want to instruct
 mirror maker where to consumer data from.

 On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma kumarsharma.ni...@gmail.com
 
 wrote:

  thanks Mayuresh and Jiangjie for your response.
 
  I have actually not understood Mirror maker clearly and hence bit
 skeptical
  if i will be able to execute it effectively.
 
  Online i hv seen the following command to be execute, but not understood
   what is consumer-1  -2.properties here? do i need to copy from my
  consumer code? also, any reason why to provide consumer property?
 
  bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
  consumer-1.properties --consumer.config consumer-2.properties
  --producer.config producer.properties --whitelist my-topic
 
 
  Regards,
  Nitin Kumar Sharma.
 
 
  On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat 
  gharatmayures...@gmail.com
   wrote:
 
   You can use the Mirror maker to move data from one data center to other
  and
   once all the data has been moved you can shut down the source data
 center
   by doing a controlled shutdown.
  
   Thanks,
  
   Mayuresh
  
   On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin
 j...@linkedin.com.invalid
  
   wrote:
  
If you want to do a seamless migration. I think a better way is to
  build
   a
cross datacenter Kafka cluster temporarily. So the process is:
1. Add several new Kafka brokers in your new datacenter and add them
 to
the old cluster.
2. Use replica assignment tool to reassign all the partitions to
  brokers
in new datacenter.
3. Perform controlled shutdown on the brokers in old datacenter.
   
Jiangjie (Becket) Qin
   
On 3/25/15, 2:01 PM, nitin sharma kumarsharma.ni...@gmail.com
  wrote:
   
Hi Team,

in my project, we have built a new datacenter for Kafka brokers and
   wants
to migrate from current datacenter to new one.

Switching producers and consumers wont be a problem provided New
Datacenter
has all the messages of existing Datacenter.


i have only 1 topic with 2 partition that need to be migrated...
 that
   too
it is only 1 time activity.

Kindly suggest the best way to deal with this situation.


Regards,
Nitin Kumar Sharma.
   
   
  
  
   --
   -Regards,
   Mayuresh R. Gharat
   (862) 250-7125
  
 



 --
 Regards,
 Tao

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect 
Qenv02zk01:2181  --topic testtopic --group KafkaMaker
Group   Topic  Pid Offset  logSize  
   Lag Owner
Exception in thread main org.I0Itec.zkclient.exception.ZkNoNodeException: 
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode 
for /consumers/KafkaMaker/offsets/testtopic/0
at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766)
at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761)
at kafka.utils.ZkUtils$.readData(ZkUtils.scala:461)
at 
kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59)
at 

Re: Kafka server relocation

[nitin sharma]

thanks Mayuresh and Jiangjie for your response.

I have actually not understood Mirror maker clearly and hence bit skeptical
if i will be able to execute it effectively.

Online i hv seen the following command to be execute, but not understood
 what is consumer-1  -2.properties here? do i need to copy from my
consumer code? also, any reason why to provide consumer property?

bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
consumer-1.properties --consumer.config consumer-2.properties
--producer.config producer.properties --whitelist my-topic


Regards,
Nitin Kumar Sharma.


On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat gharatmayures...@gmail.com
 wrote:

 You can use the Mirror maker to move data from one data center to other and
 once all the data has been moved you can shut down the source data center
 by doing a controlled shutdown.

 Thanks,

 Mayuresh

 On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin j...@linkedin.com.invalid
 wrote:

  If you want to do a seamless migration. I think a better way is to build
 a
  cross datacenter Kafka cluster temporarily. So the process is:
  1. Add several new Kafka brokers in your new datacenter and add them to
  the old cluster.
  2. Use replica assignment tool to reassign all the partitions to brokers
  in new datacenter.
  3. Perform controlled shutdown on the brokers in old datacenter.
 
  Jiangjie (Becket) Qin
 
  On 3/25/15, 2:01 PM, nitin sharma kumarsharma.ni...@gmail.com wrote:
 
  Hi Team,
  
  in my project, we have built a new datacenter for Kafka brokers and
 wants
  to migrate from current datacenter to new one.
  
  Switching producers and consumers wont be a problem provided New
  Datacenter
  has all the messages of existing Datacenter.
  
  
  i have only 1 topic with 2 partition that need to be migrated... that
 too
  it is only 1 time activity.
  
  Kindly suggest the best way to deal with this situation.
  
  
  Regards,
  Nitin Kumar Sharma.
 
 


 --
 -Regards,
 Mayuresh R. Gharat
 (862) 250-7125

Re: Kafka server relocation

[tao xiao]

Both consumer-1 and consumer-2 are properties of source clusters mirror
maker transfers data from. Mirror maker is designed to be able to consume
data from N sources (N = 1) and transfer data to one destination cluster.
You are free to supply as many consumer properties as you want to instruct
mirror maker where to consumer data from.

On Thu, Mar 26, 2015 at 9:50 PM, nitin sharma kumarsharma.ni...@gmail.com
wrote:

 thanks Mayuresh and Jiangjie for your response.

 I have actually not understood Mirror maker clearly and hence bit skeptical
 if i will be able to execute it effectively.

 Online i hv seen the following command to be execute, but not understood
  what is consumer-1  -2.properties here? do i need to copy from my
 consumer code? also, any reason why to provide consumer property?

 bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
 consumer-1.properties --consumer.config consumer-2.properties
 --producer.config producer.properties --whitelist my-topic


 Regards,
 Nitin Kumar Sharma.


 On Wed, Mar 25, 2015 at 8:57 PM, Mayuresh Gharat 
 gharatmayures...@gmail.com
  wrote:

  You can use the Mirror maker to move data from one data center to other
 and
  once all the data has been moved you can shut down the source data center
  by doing a controlled shutdown.
 
  Thanks,
 
  Mayuresh
 
  On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin j...@linkedin.com.invalid
 
  wrote:
 
   If you want to do a seamless migration. I think a better way is to
 build
  a
   cross datacenter Kafka cluster temporarily. So the process is:
   1. Add several new Kafka brokers in your new datacenter and add them to
   the old cluster.
   2. Use replica assignment tool to reassign all the partitions to
 brokers
   in new datacenter.
   3. Perform controlled shutdown on the brokers in old datacenter.
  
   Jiangjie (Becket) Qin
  
   On 3/25/15, 2:01 PM, nitin sharma kumarsharma.ni...@gmail.com
 wrote:
  
   Hi Team,
   
   in my project, we have built a new datacenter for Kafka brokers and
  wants
   to migrate from current datacenter to new one.
   
   Switching producers and consumers wont be a problem provided New
   Datacenter
   has all the messages of existing Datacenter.
   
   
   i have only 1 topic with 2 partition that need to be migrated... that
  too
   it is only 1 time activity.
   
   Kindly suggest the best way to deal with this situation.
   
   
   Regards,
   Nitin Kumar Sharma.
  
  
 
 
  --
  -Regards,
  Mayuresh R. Gharat
  (862) 250-7125
 




-- 
Regards,
Tao

Re: Kafka server relocation

[Jiangjie Qin]

If you want to do a seamless migration. I think a better way is to build a
cross datacenter Kafka cluster temporarily. So the process is:
1. Add several new Kafka brokers in your new datacenter and add them to
the old cluster.
2. Use replica assignment tool to reassign all the partitions to brokers
in new datacenter.
3. Perform controlled shutdown on the brokers in old datacenter.

Jiangjie (Becket) Qin

On 3/25/15, 2:01 PM, nitin sharma kumarsharma.ni...@gmail.com wrote:

Hi Team,

in my project, we have built a new datacenter for Kafka brokers and wants
to migrate from current datacenter to new one.

Switching producers and consumers wont be a problem provided New
Datacenter
has all the messages of existing Datacenter.


i have only 1 topic with 2 partition that need to be migrated... that too
it is only 1 time activity.

Kindly suggest the best way to deal with this situation.


Regards,
Nitin Kumar Sharma.

Re: Kafka server relocation

[Mayuresh Gharat]

You can use the Mirror maker to move data from one data center to other and
once all the data has been moved you can shut down the source data center
by doing a controlled shutdown.

Thanks,

Mayuresh

On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin j...@linkedin.com.invalid
wrote:

 If you want to do a seamless migration. I think a better way is to build a
 cross datacenter Kafka cluster temporarily. So the process is:
 1. Add several new Kafka brokers in your new datacenter and add them to
 the old cluster.
 2. Use replica assignment tool to reassign all the partitions to brokers
 in new datacenter.
 3. Perform controlled shutdown on the brokers in old datacenter.

 Jiangjie (Becket) Qin

 On 3/25/15, 2:01 PM, nitin sharma kumarsharma.ni...@gmail.com wrote:

 Hi Team,
 
 in my project, we have built a new datacenter for Kafka brokers and wants
 to migrate from current datacenter to new one.
 
 Switching producers and consumers wont be a problem provided New
 Datacenter
 has all the messages of existing Datacenter.
 
 
 i have only 1 topic with 2 partition that need to be migrated... that too
 it is only 1 time activity.
 
 Kindly suggest the best way to deal with this situation.
 
 
 Regards,
 Nitin Kumar Sharma.




-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125