Hi Neha, It turned out to be a Java Heap error. The try catch wasn't implemented properly.
Regards, Samir On Tue, Jul 23, 2013 at 11:20 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Samir, > > When you observe that the consumer hangs, could you send around the thread > dump. Also can you check the value of the FetchRate mbean ? > > Thanks, > Neha > > > On Tue, Jul 23, 2013 at 10:17 AM, Samir Madhavan < > samir.madha...@fluturasolutions.com> wrote: > > > We just observed that the consumer is occupying almost complete RAM which > > is 4Gb. > > > > > > On Tue, Jul 23, 2013 at 10:25 PM, Samir Madhavan < > > samir.madha...@fluturasolutions.com> wrote: > > > > > One of the common pattern we observed when we ran consumer from start > > > multiple times is that it catches up till the data that producer has > > > produced and then it just gets hung. If we kill it and then start it > > again, > > > it starts consuming and then again it just gets hung. We have added the > > try > > > catch statements but there seems to be no error coming. > > > > > > > > > On Tue, Jul 23, 2013 at 9:06 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > >> Not sure about the error. However, your consumer seems to be lagging. > > Are > > >> the consumer offsets moving at all? If not, are your consumer threads > > >> still > > >> alive? You probably want to try/catch the consumer code to see if > there > > is > > >> any unexpected exceptions. > > >> > > >> Thanks, > > >> > > >> Jun > > >> > > >> > > >> On Tue, Jul 23, 2013 at 7:58 AM, Anurup Raveendran < > > >> anurup.raveend...@fluturasolutions.com> wrote: > > >> > > >> > After consuming about 5 gb of messages , it stopped consuming and it > > got > > >> > stuck at a particular offset. After running the following command I > > got > > >> an > > >> > error regarding the broker. > > >> > > > >> > > > >> > > > >> > > > >> > $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > > >> > air-DummyProductionConsumerGroup-After --zkconnect > 172.16.18.65:2181 > > >> --topic > > >> > air > > >> > > > >> > > > >> > > > >> > [2013-07-23 20:19:42,748] INFO Starting ZkClient event thread. > > >> > (org.I0Itec.zkclient.ZkEventThread) > > >> > [2013-07-23 20:19:42,757] INFO Client > > >> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 > 05:47 > > >> GMT > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,757] INFO Client environment:host.name > =localhost > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,757] INFO Client > > environment:java.version=1.7.0_21 > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,757] INFO Client environment:java.vendor=Oracle > > >> > Corporation (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,757] INFO Client > > >> > environment:java.home=/usr/java/jdk1.7.0_21/jre > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,757] INFO Client > > >> > > > >> > > > >> > > > environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client > > >> > > > >> > > > >> > > > environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client > environment:java.io.tmpdir=/tmp > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client environment:java.compiler=<NA> > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client environment:os.name=Linux > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client environment:os.arch=amd64 > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client > > >> > environment:os.version=2.6.32-279.22.1.el6.x86_64 > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client environment:user.name > =cleartrip > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client > > >> environment:user.home=/home/cleartrip > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,758] INFO Client > > >> > > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,759] INFO Initiating client connection, > > >> connectString= > > >> > 172.16.18.65:2181 sessionTimeout=30000 > > >> > watcher=org.I0Itec.zkclient.ZkClient@6926e27 > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:19:42,783] INFO Opening socket connection to server / > > >> > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) > > >> > [2013-07-23 20:19:42,788] INFO Socket connection established to > zoo1/ > > >> > 172.16.18.65:2181, initiating session > > (org.apache.zookeeper.ClientCnxn) > > >> > [2013-07-23 20:19:42,800] INFO Session establishment complete on > > server > > >> > zoo1/172.16.18.65:2181, sessionid = 0x13fc7235423015f, negotiated > > >> timeout > > >> > = > > >> > 30000 (org.apache.zookeeper.ClientCnxn) > > >> > [2013-07-23 20:19:42,802] INFO zookeeper state changed > (SyncConnected) > > >> > (org.I0Itec.zkclient.ZkClient) > > >> > air-DummyProductionConsumerGroup-After,air,1-0 > > >> > (Group,Topic,BrokerId-PartitionId) > > >> > Owner = > > >> > > > >> > > > >> > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 > > >> > Consumer offset = 1140946916 > > >> > = 1,140,946,916 (1.06G) > > >> > [2013-07-23 20:19:42,948] ERROR Could not parse broker info > > >> > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 > > >> > (kafka.tools.ConsumerOffsetChecker$) > > >> > air-DummyProductionConsumerGroup-After,air,1-1 > > >> > (Group,Topic,BrokerId-PartitionId) > > >> > Owner = > > >> > > > >> > > > >> > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 > > >> > Consumer offset = 1142891729 > > >> > = 1,142,891,729 (1.06G) > > >> > air-DummyProductionConsumerGroup-After,air,2-0 > > >> > (Group,Topic,BrokerId-PartitionId) > > >> > Owner = > > >> > > > >> > > > >> > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 > > >> > Consumer offset = 1587266466 > > >> > = 1,587,266,466 (1.48G) > > >> > [2013-07-23 20:19:42,956] ERROR Could not parse broker info > > >> > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 > > >> > (kafka.tools.ConsumerOffsetChecker$) > > >> > air-DummyProductionConsumerGroup-After,air,2-1 > > >> > (Group,Topic,BrokerId-PartitionId) > > >> > Owner = > > >> > > > >> > > > >> > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 > > >> > Consumer offset = 1566027800 > > >> > = 1,566,027,800 (1.46G) > > >> > BROKER INFO > > >> > [2013-07-23 20:19:42,988] INFO Terminate ZkClient event thread. > > >> > (org.I0Itec.zkclient.ZkEventThread) > > >> > [2013-07-23 20:19:42,992] INFO EventThread shut down > > >> > (org.apache.zookeeper.ClientCnxn) > > >> > [2013-07-23 20:19:42,992] INFO Session: 0x13fc7235423015f closed > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ clear > > >> > > > >> > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ > > >> > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > > >> > air-DummyProductionConsumerGroup-After --zkconnect > 172.16.18.65:2181 > > >> --topic > > >> > air > > >> > [2013-07-23 20:20:12,669] INFO Starting ZkClient event thread. > > >> > (org.I0Itec.zkclient.ZkEventThread) > > >> > [2013-07-23 20:20:12,679] INFO Client > > >> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 > 05:47 > > >> GMT > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,679] INFO Client environment:host.name > =localhost > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,679] INFO Client > > environment:java.version=1.7.0_21 > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,679] INFO Client environment:java.vendor=Oracle > > >> > Corporation (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,679] INFO Client > > >> > environment:java.home=/usr/java/jdk1.7.0_21/jre > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,679] INFO Client > > >> > > > >> > > > >> > > > environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,679] INFO Client > > >> > > > >> > > > >> > > > environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,680] INFO Client > environment:java.io.tmpdir=/tmp > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,680] INFO Client environment:java.compiler=<NA> > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,680] INFO Client environment:os.name=Linux > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,680] INFO Client environment:os.arch=amd64 > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,680] INFO Client > > >> > environment:os.version=2.6.32-279.22.1.el6.x86_64 > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,680] INFO Client environment:user.name > =cleartrip > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,680] INFO Client > > >> environment:user.home=/home/cleartrip > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,680] INFO Client > > >> > > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,681] INFO Initiating client connection, > > >> connectString= > > >> > 172.16.18.65:2181 sessionTimeout=30000 > > >> > watcher=org.I0Itec.zkclient.ZkClient@6926e27 > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > [2013-07-23 20:20:12,705] INFO Opening socket connection to server / > > >> > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) > > >> > [2013-07-23 20:20:12,710] INFO Socket connection established to > zoo1/ > > >> > 172.16.18.65:2181, initiating session > > (org.apache.zookeeper.ClientCnxn) > > >> > [2013-07-23 20:20:12,719] INFO Session establishment complete on > > server > > >> > zoo1/172.16.18.65:2181, sessionid = 0x13fc72354230160, negotiated > > >> timeout > > >> > = > > >> > 30000 (org.apache.zookeeper.ClientCnxn) > > >> > [2013-07-23 20:20:12,722] INFO zookeeper state changed > (SyncConnected) > > >> > (org.I0Itec.zkclient.ZkClient) > > >> > air-DummyProductionConsumerGroup-After,air,1-0 > > >> > (Group,Topic,BrokerId-PartitionId) > > >> > Owner = > > >> > > > >> > > > >> > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 > > >> > Consumer offset = 1140946916 > > >> > = 1,140,946,916 (1.06G) > > >> > [2013-07-23 20:20:12,828] ERROR Could not parse broker info > > >> > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 > > >> > (kafka.tools.ConsumerOffsetChecker$) > > >> > air-DummyProductionConsumerGroup-After,air,1-1 > > >> > (Group,Topic,BrokerId-PartitionId) > > >> > Owner = > > >> > > > >> > > > >> > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 > > >> > Consumer offset = 1142891729 > > >> > = 1,142,891,729 (1.06G) > > >> > air-DummyProductionConsumerGroup-After,air,2-0 > > >> > (Group,Topic,BrokerId-PartitionId) > > >> > Owner = > > >> > > > >> > > > >> > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 > > >> > Consumer offset = 1587266466 > > >> > = 1,587,266,466 (1.48G) > > >> > [2013-07-23 20:20:12,835] ERROR Could not parse broker info > > >> > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 > > >> > (kafka.tools.ConsumerOffsetChecker$) > > >> > air-DummyProductionConsumerGroup-After,air,2-1 > > >> > (Group,Topic,BrokerId-PartitionId) > > >> > Owner = > > >> > > > >> > > > >> > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 > > >> > Consumer offset = 1566027800 > > >> > = 1,566,027,800 (1.46G) > > >> > BROKER INFO > > >> > [2013-07-23 20:20:12,867] INFO Terminate ZkClient event thread. > > >> > (org.I0Itec.zkclient.ZkEventThread) > > >> > [2013-07-23 20:20:12,871] INFO EventThread shut down > > >> > (org.apache.zookeeper.ClientCnxn) > > >> > [2013-07-23 20:20:12,871] INFO Session: 0x13fc72354230160 closed > > >> > (org.apache.zookeeper.ZooKeeper) > > >> > > > >> > > > > > > > > > > > > -- > > > *Samir Madhavan *| Data Scientist | Flutura Business Solutions Pvt. > Ltd | > > > 4th Floor, 'Geetanjali', #693, 15th Cross, J.P Nagar 2nd Phase, > > > Bangalore, India - 560078 | Mobile: +91 9886139631 | email: * > > > samir.madha...@fluturasolutions.com*| www.fluturasolutions.com | > > > > > > > > > > > -- > > *Samir Madhavan *| Data Scientist | Flutura Business Solutions Pvt. Ltd | > > 4th > > Floor, 'Geetanjali', #693, 15th Cross, J.P Nagar 2nd Phase, Bangalore, > > India - 560078 | Mobile: +91 9886139631 | email: * > > samir.madha...@fluturasolutions.com*| www.fluturasolutions.com | > > > -- *Samir Madhavan *| Data Scientist | Flutura Business Solutions Pvt. Ltd | 4th Floor, 'Geetanjali', #693, 15th Cross, J.P Nagar 2nd Phase, Bangalore, India - 560078 | Mobile: +91 9886139631 | email: * samir.madha...@fluturasolutions.com*| www.fluturasolutions.com |
