Samir, When you observe that the consumer hangs, could you send around the thread dump. Also can you check the value of the FetchRate mbean ?
Thanks, Neha On Tue, Jul 23, 2013 at 10:17 AM, Samir Madhavan < samir.madha...@fluturasolutions.com> wrote: > We just observed that the consumer is occupying almost complete RAM which > is 4Gb. > > > On Tue, Jul 23, 2013 at 10:25 PM, Samir Madhavan < > samir.madha...@fluturasolutions.com> wrote: > > > One of the common pattern we observed when we ran consumer from start > > multiple times is that it catches up till the data that producer has > > produced and then it just gets hung. If we kill it and then start it > again, > > it starts consuming and then again it just gets hung. We have added the > try > > catch statements but there seems to be no error coming. > > > > > > On Tue, Jul 23, 2013 at 9:06 PM, Jun Rao <jun...@gmail.com> wrote: > > > >> Not sure about the error. However, your consumer seems to be lagging. > Are > >> the consumer offsets moving at all? If not, are your consumer threads > >> still > >> alive? You probably want to try/catch the consumer code to see if there > is > >> any unexpected exceptions. > >> > >> Thanks, > >> > >> Jun > >> > >> > >> On Tue, Jul 23, 2013 at 7:58 AM, Anurup Raveendran < > >> anurup.raveend...@fluturasolutions.com> wrote: > >> > >> > After consuming about 5 gb of messages , it stopped consuming and it > got > >> > stuck at a particular offset. After running the following command I > got > >> an > >> > error regarding the broker. > >> > > >> > > >> > > >> > > >> > $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > >> > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181 > >> --topic > >> > air > >> > > >> > > >> > > >> > [2013-07-23 20:19:42,748] INFO Starting ZkClient event thread. > >> > (org.I0Itec.zkclient.ZkEventThread) > >> > [2013-07-23 20:19:42,757] INFO Client > >> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 > >> GMT > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,757] INFO Client environment:host.name=localhost > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,757] INFO Client > environment:java.version=1.7.0_21 > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,757] INFO Client environment:java.vendor=Oracle > >> > Corporation (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,757] INFO Client > >> > environment:java.home=/usr/java/jdk1.7.0_21/jre > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,757] INFO Client > >> > > >> > > >> > environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client > >> > > >> > > >> > environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client environment:java.io.tmpdir=/tmp > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client environment:java.compiler=<NA> > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client environment:os.name=Linux > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client environment:os.arch=amd64 > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client > >> > environment:os.version=2.6.32-279.22.1.el6.x86_64 > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client environment:user.name=cleartrip > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client > >> environment:user.home=/home/cleartrip > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,758] INFO Client > >> > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,759] INFO Initiating client connection, > >> connectString= > >> > 172.16.18.65:2181 sessionTimeout=30000 > >> > watcher=org.I0Itec.zkclient.ZkClient@6926e27 > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:19:42,783] INFO Opening socket connection to server / > >> > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) > >> > [2013-07-23 20:19:42,788] INFO Socket connection established to zoo1/ > >> > 172.16.18.65:2181, initiating session > (org.apache.zookeeper.ClientCnxn) > >> > [2013-07-23 20:19:42,800] INFO Session establishment complete on > server > >> > zoo1/172.16.18.65:2181, sessionid = 0x13fc7235423015f, negotiated > >> timeout > >> > = > >> > 30000 (org.apache.zookeeper.ClientCnxn) > >> > [2013-07-23 20:19:42,802] INFO zookeeper state changed (SyncConnected) > >> > (org.I0Itec.zkclient.ZkClient) > >> > air-DummyProductionConsumerGroup-After,air,1-0 > >> > (Group,Topic,BrokerId-PartitionId) > >> > Owner = > >> > > >> > > >> > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 > >> > Consumer offset = 1140946916 > >> > = 1,140,946,916 (1.06G) > >> > [2013-07-23 20:19:42,948] ERROR Could not parse broker info > >> > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 > >> > (kafka.tools.ConsumerOffsetChecker$) > >> > air-DummyProductionConsumerGroup-After,air,1-1 > >> > (Group,Topic,BrokerId-PartitionId) > >> > Owner = > >> > > >> > > >> > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 > >> > Consumer offset = 1142891729 > >> > = 1,142,891,729 (1.06G) > >> > air-DummyProductionConsumerGroup-After,air,2-0 > >> > (Group,Topic,BrokerId-PartitionId) > >> > Owner = > >> > > >> > > >> > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 > >> > Consumer offset = 1587266466 > >> > = 1,587,266,466 (1.48G) > >> > [2013-07-23 20:19:42,956] ERROR Could not parse broker info > >> > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 > >> > (kafka.tools.ConsumerOffsetChecker$) > >> > air-DummyProductionConsumerGroup-After,air,2-1 > >> > (Group,Topic,BrokerId-PartitionId) > >> > Owner = > >> > > >> > > >> > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 > >> > Consumer offset = 1566027800 > >> > = 1,566,027,800 (1.46G) > >> > BROKER INFO > >> > [2013-07-23 20:19:42,988] INFO Terminate ZkClient event thread. > >> > (org.I0Itec.zkclient.ZkEventThread) > >> > [2013-07-23 20:19:42,992] INFO EventThread shut down > >> > (org.apache.zookeeper.ClientCnxn) > >> > [2013-07-23 20:19:42,992] INFO Session: 0x13fc7235423015f closed > >> > (org.apache.zookeeper.ZooKeeper) > >> > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ clear > >> > > >> > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ > >> > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > >> > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181 > >> --topic > >> > air > >> > [2013-07-23 20:20:12,669] INFO Starting ZkClient event thread. > >> > (org.I0Itec.zkclient.ZkEventThread) > >> > [2013-07-23 20:20:12,679] INFO Client > >> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 > >> GMT > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,679] INFO Client environment:host.name=localhost > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,679] INFO Client > environment:java.version=1.7.0_21 > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,679] INFO Client environment:java.vendor=Oracle > >> > Corporation (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,679] INFO Client > >> > environment:java.home=/usr/java/jdk1.7.0_21/jre > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,679] INFO Client > >> > > >> > > >> > environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,679] INFO Client > >> > > >> > > >> > environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,680] INFO Client environment:java.io.tmpdir=/tmp > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,680] INFO Client environment:java.compiler=<NA> > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,680] INFO Client environment:os.name=Linux > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,680] INFO Client environment:os.arch=amd64 > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,680] INFO Client > >> > environment:os.version=2.6.32-279.22.1.el6.x86_64 > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,680] INFO Client environment:user.name=cleartrip > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,680] INFO Client > >> environment:user.home=/home/cleartrip > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,680] INFO Client > >> > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,681] INFO Initiating client connection, > >> connectString= > >> > 172.16.18.65:2181 sessionTimeout=30000 > >> > watcher=org.I0Itec.zkclient.ZkClient@6926e27 > >> > (org.apache.zookeeper.ZooKeeper) > >> > [2013-07-23 20:20:12,705] INFO Opening socket connection to server / > >> > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) > >> > [2013-07-23 20:20:12,710] INFO Socket connection established to zoo1/ > >> > 172.16.18.65:2181, initiating session > (org.apache.zookeeper.ClientCnxn) > >> > [2013-07-23 20:20:12,719] INFO Session establishment complete on > server > >> > zoo1/172.16.18.65:2181, sessionid = 0x13fc72354230160, negotiated > >> timeout > >> > = > >> > 30000 (org.apache.zookeeper.ClientCnxn) > >> > [2013-07-23 20:20:12,722] INFO zookeeper state changed (SyncConnected) > >> > (org.I0Itec.zkclient.ZkClient) > >> > air-DummyProductionConsumerGroup-After,air,1-0 > >> > (Group,Topic,BrokerId-PartitionId) > >> > Owner = > >> > > >> > > >> > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 > >> > Consumer offset = 1140946916 > >> > = 1,140,946,916 (1.06G) > >> > [2013-07-23 20:20:12,828] ERROR Could not parse broker info > >> > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 > >> > (kafka.tools.ConsumerOffsetChecker$) > >> > air-DummyProductionConsumerGroup-After,air,1-1 > >> > (Group,Topic,BrokerId-PartitionId) > >> > Owner = > >> > > >> > > >> > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 > >> > Consumer offset = 1142891729 > >> > = 1,142,891,729 (1.06G) > >> > air-DummyProductionConsumerGroup-After,air,2-0 > >> > (Group,Topic,BrokerId-PartitionId) > >> > Owner = > >> > > >> > > >> > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 > >> > Consumer offset = 1587266466 > >> > = 1,587,266,466 (1.48G) > >> > [2013-07-23 20:20:12,835] ERROR Could not parse broker info > >> > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 > >> > (kafka.tools.ConsumerOffsetChecker$) > >> > air-DummyProductionConsumerGroup-After,air,2-1 > >> > (Group,Topic,BrokerId-PartitionId) > >> > Owner = > >> > > >> > > >> > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 > >> > Consumer offset = 1566027800 > >> > = 1,566,027,800 (1.46G) > >> > BROKER INFO > >> > [2013-07-23 20:20:12,867] INFO Terminate ZkClient event thread. > >> > (org.I0Itec.zkclient.ZkEventThread) > >> > [2013-07-23 20:20:12,871] INFO EventThread shut down > >> > (org.apache.zookeeper.ClientCnxn) > >> > [2013-07-23 20:20:12,871] INFO Session: 0x13fc72354230160 closed > >> > (org.apache.zookeeper.ZooKeeper) > >> > > >> > > > > > > > > -- > > *Samir Madhavan *| Data Scientist | Flutura Business Solutions Pvt. Ltd | > > 4th Floor, 'Geetanjali', #693, 15th Cross, J.P Nagar 2nd Phase, > > Bangalore, India - 560078 | Mobile: +91 9886139631 | email: * > > samir.madha...@fluturasolutions.com*| www.fluturasolutions.com | > > > > > > -- > *Samir Madhavan *| Data Scientist | Flutura Business Solutions Pvt. Ltd | > 4th > Floor, 'Geetanjali', #693, 15th Cross, J.P Nagar 2nd Phase, Bangalore, > India - 560078 | Mobile: +91 9886139631 | email: * > samir.madha...@fluturasolutions.com*| www.fluturasolutions.com | >
