Re: Consumer stops consuming after 5 gb

2013-07-29 Thread Samir Madhavan
Hi Neha, It turned out to be a Java Heap error. The try catch wasn't implemented properly. Regards, Samir On Tue, Jul 23, 2013 at 11:20 PM, Neha Narkhede wrote: > Samir, > > When you observe that the consumer hangs, could you send around the thread > dump. Also can you check the value of the F

Re: Consumer stops consuming after 5 gb

2013-07-23 Thread Neha Narkhede
Samir, When you observe that the consumer hangs, could you send around the thread dump. Also can you check the value of the FetchRate mbean ? Thanks, Neha On Tue, Jul 23, 2013 at 10:17 AM, Samir Madhavan < samir.madha...@fluturasolutions.com> wrote: > We just observed that the consumer is occu

Re: Consumer stops consuming after 5 gb

2013-07-23 Thread Samir Madhavan
We just observed that the consumer is occupying almost complete RAM which is 4Gb. On Tue, Jul 23, 2013 at 10:25 PM, Samir Madhavan < samir.madha...@fluturasolutions.com> wrote: > One of the common pattern we observed when we ran consumer from start > multiple times is that it catches up till the

Re: Consumer stops consuming after 5 gb

2013-07-23 Thread Samir Madhavan
One of the common pattern we observed when we ran consumer from start multiple times is that it catches up till the data that producer has produced and then it just gets hung. If we kill it and then start it again, it starts consuming and then again it just gets hung. We have added the try catch st

Re: Consumer stops consuming after 5 gb

2013-07-23 Thread Jun Rao
Not sure about the error. However, your consumer seems to be lagging. Are the consumer offsets moving at all? If not, are your consumer threads still alive? You probably want to try/catch the consumer code to see if there is any unexpected exceptions. Thanks, Jun On Tue, Jul 23, 2013 at 7:58 AM

Consumer stops consuming after 5 gb

2013-07-23 Thread Anurup Raveendran
After consuming about 5 gb of messages , it stopped consuming and it got stuck at a particular offset. After running the following command I got an error regarding the broker. $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group air-DummyProductionConsumerGroup-After --zkconnect 1