Delay in fetching messages - High level consumer -Kafka 0.8

2014-02-25 Thread Arjun
Hi, I am using kafka 0.8. I have 3 brokers on three systems and 3 zookeepers running. I am using the high level consumer which is in examples folder of kafka. I am able to push the messages into the queue, but retriving the messages is taking some time. Is there any way i can tune this. I

Re: Delay in fetching messages - High level consumer -Kafka 0.8

2014-02-25 Thread Arjun
Apart from that i get this stack trace 25 Feb 2014 15:45:22,636 WARN [ConsumerFetcherThread-group1_www.taf-dev.com-1393322165136-8318b07d-0-0] [kafka.consumer.ConsumerFetcherThread] [ConsumerFetcherThread-group1_www.taf-dev.com-1393322165136-8318b07d-0-0], Error in fetch Name: FetchRequest;

Re: Delay in fetching messages - High level consumer -Kafka 0.8

2014-02-25 Thread Arjun
Adding to this, i have started my logs in trace mode. I fount that the Consumer fetcher threads are sending the meta data but are not receiving any. I see all the TRACE [ConsumerFetcherThread-group1_www.taf-dev.com-1393329622308-6e15dd12-0-1] [kafka.network.BoundedByteBufferSend] 205 bytes

Re: Delay in fetching messages - High level consumer -Kafka 0.8

2014-02-25 Thread Arjun
The thing i found is my ConsumerFetcherThreads are not going beyond BoundedByteBufferReceive.readFrom. When i added a few more traces in that function i found that the call is stalling after exceptIncomplete function. I guess Utils.read is stalling for more than 30 sec, which is the socket

Re: Delay in fetching messages - High level consumer -Kafka 0.8

2014-02-25 Thread Neha Narkhede
Arjun, Have you looked at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why ? Thanks, Neha On Tue, Feb 25, 2014 at 5:04 AM, Arjun ar...@socialtwist.com wrote: The thing i found is my ConsumerFetcherThreads are not going beyond

Re: Delay in fetching messages - High level consumer -Kafka 0.8

2014-02-25 Thread Jun Rao
The following config is probably what's causing the socket timeout. Try sth like 1000ms. MaxWait: 1000 ms Thanks, Jun On Tue, Feb 25, 2014 at 2:16 AM, Arjun ar...@socialtwist.com wrote: Apart from that i get this stack trace 25 Feb 2014 15:45:22,636 WARN [ConsumerFetcherThread-group1_

Re: Delay in fetching messages - High level consumer -Kafka 0.8

2014-02-25 Thread Arjun
Hi, As i have mentioned in the first message, I have checked the log and offset using the Consumer off set checker tool. The Consumer offset just stalls. And there is a lag. I haven't specified any fetch size in the consumer so i guess there is a default size of 1MB. All my messages are less

Re: Delay in fetching messages - High level consumer -Kafka 0.8

2014-02-25 Thread Arjun
Hi I will make the change and see whether things work fine or not and let you know. Thanks Arjun Narasimha Kota On Tuesday 25 February 2014 09:58 PM, Jun Rao wrote: The following config is probably what's causing the socket timeout. Try sth like 1000ms. MaxWait: 1000 ms Thanks, Jun