Re: Kafka consumer configuration / performance issues

Sorry to bump this up, can anyone provide some input on this ? I need to make a call soon whether kafka is a good fit to our requirement. On Tuesday, October 4, 2016 8:57 PM, Shamik Banerjee wrote: Hi, I'm a newbie trying out kafka as an alternative to AWS SQS.

Kafka consumer configuration / performance issues

Hi, I'm a newbie trying out kafka as an alternative to AWS SQS. The motivation primarily is to improve performance where kafka would eliminate the constraint of pulling 10 messages at a time with a cap of 256kb. Here's a high-level scenario of my use case. I've a bunch of crawlers which are

Tools/recommendations to debug performance issues?

We're using 0.8.2.1 processing maybe 1 million messages per hour. Each message includes tracking information with a timestamp for when it was produced, and a timestamp for when it was consumed, to give us roughly the amount of time it spent in Kafka. On average this number is in the seconds and

Re: Tools/recommendations to debug performance issues?

Have you checked the consumer lag? You can use the offset checker tool to see if there is a lag. On 14 Sep 2015 18:36, "noah" wrote: > We're using 0.8.2.1 processing maybe 1 million messages per hour. Each > message includes tracking information with a timestamp for when it

Re: Tools/recommendations to debug performance issues?

Kafka also collects very useful metrics on request times and their breakdown. They are under kafka.network. On Mon, Sep 14, 2015 at 6:59 AM, Rahul Jain wrote: > Have you checked the consumer lag? You can use the offset checker tool to > see if there is a lag. > On 14 Sep

Re: Performance issues

By increasing partitions and using kafka from master branch I was able to cut down the response times into half. But it still seems high and it looks like there still is a delay between a successful post and the first time message is seen by the consumers. There are plenty of resources available.

Re: Performance issues

I can't find this property in server.properties file. Is that the right place to set this parameter? On Tue, Oct 21, 2014 at 6:27 PM, Jun Rao jun...@gmail.com wrote: Could you also set replica.fetch.wait.max.ms in the broker to sth much smaller? Thanks, Jun On Tue, Oct 21, 2014 at 2:15

Re: Performance issues

I have a java test that produces messages and then consumer consumers it. Consumers are active all the time. There is 1 consumer for 1 producer. I am measuring the time between the message is successfully written to the queue and the time consumer picks it up. On Tue, Oct 21, 2014 at 8:32 AM, Neha

Re: Performance issues

This is the version I am using: kafka_2.10-0.8.1.1 I think this is fairly recent version On Tue, Oct 21, 2014 at 10:57 AM, Jay Kreps jay.kr...@gmail.com wrote: What version of Kafka is this? Can you try the same test against trunk? We fixed a couple of latency related bugs which may be the

Re: Performance issues

There was a bug that could lead to the fetch request from the consumer hitting it's timeout instead of being immediately triggered by the produce request. To see if you are effected by that set you consumer max wait time to 1 ms and see if the latency drops to 1 ms (or, alternately, try with trunk

Re: Performance issues

Is this a parameter I need to set it in kafka server or on the client side? Also, can you help point out which one exactly is consumer max wait time from this list? https://kafka.apache.org/08/configuration.html On Tue, Oct 21, 2014 at 11:35 AM, Jay Kreps jay.kr...@gmail.com wrote: There was a

Re: Performance issues

This is a consumer config: fetch.wait.max.ms On Tue, Oct 21, 2014 at 11:39 AM, Mohit Anchlia mohitanch...@gmail.com wrote: Is this a parameter I need to set it in kafka server or on the client side? Also, can you help point out which one exactly is consumer max wait time from this list?

Re: Performance issues

I set the property to 1 in the consumer code that is passed to createJavaConsumerConnector code, but it didn't seem to help props.put(fetch.wait.max.ms, fetchMaxWait); On Tue, Oct 21, 2014 at 1:21 PM, Guozhang Wang wangg...@gmail.com wrote: This is a consumer config: fetch.wait.max.ms On

Re: Performance issues

Most of the consumer threads seems to be waiting: ConsumerFetcherThread-groupA_ip-10-38-19-230-1413925671158-3cc3e22f-0-0 prio=10 tid=0x7f0aa84db800 nid=0x5be9 runnable [0x7f0a5a618000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

Performance issues

I am running a performance test and from what I am seeing is that messages are taking about 100ms to pop from the queue itself and hence making the test slow. I am looking for pointers of how I can troubleshoot this issue. There seems to be plenty of CPU and IO available. I am running 22