num.consumer.fetchers means the max number of fetcher threads that can be
spawned. it doesn't necessarily mean you can get as many fetcher threads as
you specify.
To me the metrics are suggesting a very slow consumption rate only 18.21
bytes/minute. Here is the benchmark Linkedin does
hi Xiao,
i have finally got JMX monitoring enabled for my kafka nodes in test
envrionment and here is what i observed.
i was monitoring mbeans under kafka.consumer domain of JVM running Kafka
Mirror Maker process.
=
AllTopicsBytes === 18.21 bytes/minute
how about the consumer lag of mirror maker?
On Mon, Apr 13, 2015 at 1:33 PM, nitin sharma kumarsharma.ni...@gmail.com
wrote:
i just tested that too and below is the stats.. it is clear that with
kafka-consumer-perf-test.sh, i am able to get a high throughput. around
44.0213 MB/sec.
i just tested that too and below is the stats.. it is clear that with
kafka-consumer-perf-test.sh, i am able to get a high throughput. around
44.0213 MB/sec.
Seriously some configuration needs to be tweaked in MirrorMaker
configuration for speedy processing... Can you think of something ?
thanks, but can you please tell which metrics could highlight the factor
causing slow data migration by MirrorMaker?
Regards,
Nitin Kumar Sharma.
On Tue, Apr 7, 2015 at 10:10 PM, tao xiao xiaotao...@gmail.com wrote:
You may need to look into the consumer metrics and producer metrics to
Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check if
the consumer has problem processing messages
On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma kumarsharma.ni...@gmail.com
wrote:
thanks, but can you please tell which metrics could highlight the factor
causing slow data
hi,
sorry for late response. ... i have been able to fix the issue .. problem
was in my approach. I got confused between my source and target system
while defining consumer producer property file .. it is fixed now
Now new issue.. the rate at which data is migrated is very very slow... i
mean
You may need to look into the consumer metrics and producer metrics to
identify the root cause. metrics in kafka.consumer and kafka.producer
categories will help you find out the problems.
This link gives instruction how to read the metrics
http://kafka.apache.org/documentation.html#monitoring
Can you attach your mirror maker log?
On Wed, Apr 1, 2015 at 12:28 AM, nitin sharma kumarsharma.ni...@gmail.com
wrote:
i tried with auto.offset.reset=smallest, but still not working..
there is data in my source cluster
Regards,
Nitin Kumar Sharma.
On Mon, Mar 30, 2015 at 10:30 PM, tao
thanks Xiao
I tried MirrorMaker option in my test environment but failed. I am not able
to see the log getting copied to destination cluster. I see in the log of
MirrorMaker process that connection is successfully established between
source and destination cluster but still not sure what i
thanks Mayuresh and Jiangjie for your response.
I have actually not understood Mirror maker clearly and hence bit skeptical
if i will be able to execute it effectively.
Online i hv seen the following command to be execute, but not understood
what is consumer-1 -2.properties here? do i need to
Both consumer-1 and consumer-2 are properties of source clusters mirror
maker transfers data from. Mirror maker is designed to be able to consume
data from N sources (N = 1) and transfer data to one destination cluster.
You are free to supply as many consumer properties as you want to instruct
If you want to do a seamless migration. I think a better way is to build a
cross datacenter Kafka cluster temporarily. So the process is:
1. Add several new Kafka brokers in your new datacenter and add them to
the old cluster.
2. Use replica assignment tool to reassign all the partitions to
Hi Team,
in my project, we have built a new datacenter for Kafka brokers and wants
to migrate from current datacenter to new one.
Switching producers and consumers wont be a problem provided New Datacenter
has all the messages of existing Datacenter.
i have only 1 topic with 2 partition that
You can use the Mirror maker to move data from one data center to other and
once all the data has been moved you can shut down the source data center
by doing a controlled shutdown.
Thanks,
Mayuresh
On Wed, Mar 25, 2015 at 2:35 PM, Jiangjie Qin j...@linkedin.com.invalid
wrote:
If you want to
15 matches
Mail list logo
