We just observed that the consumer is occupying almost complete RAM which is 4Gb.
On Tue, Jul 23, 2013 at 10:25 PM, Samir Madhavan < samir.madha...@fluturasolutions.com> wrote: > One of the common pattern we observed when we ran consumer from start > multiple times is that it catches up till the data that producer has > produced and then it just gets hung. If we kill it and then start it again, > it starts consuming and then again it just gets hung. We have added the try > catch statements but there seems to be no error coming. > > > On Tue, Jul 23, 2013 at 9:06 PM, Jun Rao <jun...@gmail.com> wrote: > >> Not sure about the error. However, your consumer seems to be lagging. Are >> the consumer offsets moving at all? If not, are your consumer threads >> still >> alive? You probably want to try/catch the consumer code to see if there is >> any unexpected exceptions. >> >> Thanks, >> >> Jun >> >> >> On Tue, Jul 23, 2013 at 7:58 AM, Anurup Raveendran < >> anurup.raveend...@fluturasolutions.com> wrote: >> >> > After consuming about 5 gb of messages , it stopped consuming and it got >> > stuck at a particular offset. After running the following command I got >> an >> > error regarding the broker. >> > >> > >> > >> > >> > $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group >> > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181 >> --topic >> > air >> > >> > >> > >> > [2013-07-23 20:19:42,748] INFO Starting ZkClient event thread. >> > (org.I0Itec.zkclient.ZkEventThread) >> > [2013-07-23 20:19:42,757] INFO Client >> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 >> GMT >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,757] INFO Client environment:host.name=localhost >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,757] INFO Client environment:java.version=1.7.0_21 >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,757] INFO Client environment:java.vendor=Oracle >> > Corporation (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,757] INFO Client >> > environment:java.home=/usr/java/jdk1.7.0_21/jre >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,757] INFO Client >> > >> > >> environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client >> > >> > >> environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client environment:java.io.tmpdir=/tmp >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client environment:java.compiler=<NA> >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client environment:os.name=Linux >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client environment:os.arch=amd64 >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client >> > environment:os.version=2.6.32-279.22.1.el6.x86_64 >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client environment:user.name=cleartrip >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client >> environment:user.home=/home/cleartrip >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,758] INFO Client >> > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,759] INFO Initiating client connection, >> connectString= >> > 172.16.18.65:2181 sessionTimeout=30000 >> > watcher=org.I0Itec.zkclient.ZkClient@6926e27 >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:19:42,783] INFO Opening socket connection to server / >> > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) >> > [2013-07-23 20:19:42,788] INFO Socket connection established to zoo1/ >> > 172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn) >> > [2013-07-23 20:19:42,800] INFO Session establishment complete on server >> > zoo1/172.16.18.65:2181, sessionid = 0x13fc7235423015f, negotiated >> timeout >> > = >> > 30000 (org.apache.zookeeper.ClientCnxn) >> > [2013-07-23 20:19:42,802] INFO zookeeper state changed (SyncConnected) >> > (org.I0Itec.zkclient.ZkClient) >> > air-DummyProductionConsumerGroup-After,air,1-0 >> > (Group,Topic,BrokerId-PartitionId) >> > Owner = >> > >> > >> air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 >> > Consumer offset = 1140946916 >> > = 1,140,946,916 (1.06G) >> > [2013-07-23 20:19:42,948] ERROR Could not parse broker info >> > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 >> > (kafka.tools.ConsumerOffsetChecker$) >> > air-DummyProductionConsumerGroup-After,air,1-1 >> > (Group,Topic,BrokerId-PartitionId) >> > Owner = >> > >> > >> air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 >> > Consumer offset = 1142891729 >> > = 1,142,891,729 (1.06G) >> > air-DummyProductionConsumerGroup-After,air,2-0 >> > (Group,Topic,BrokerId-PartitionId) >> > Owner = >> > >> > >> air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 >> > Consumer offset = 1587266466 >> > = 1,587,266,466 (1.48G) >> > [2013-07-23 20:19:42,956] ERROR Could not parse broker info >> > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 >> > (kafka.tools.ConsumerOffsetChecker$) >> > air-DummyProductionConsumerGroup-After,air,2-1 >> > (Group,Topic,BrokerId-PartitionId) >> > Owner = >> > >> > >> air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 >> > Consumer offset = 1566027800 >> > = 1,566,027,800 (1.46G) >> > BROKER INFO >> > [2013-07-23 20:19:42,988] INFO Terminate ZkClient event thread. >> > (org.I0Itec.zkclient.ZkEventThread) >> > [2013-07-23 20:19:42,992] INFO EventThread shut down >> > (org.apache.zookeeper.ClientCnxn) >> > [2013-07-23 20:19:42,992] INFO Session: 0x13fc7235423015f closed >> > (org.apache.zookeeper.ZooKeeper) >> > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ clear >> > >> > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ >> > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group >> > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181 >> --topic >> > air >> > [2013-07-23 20:20:12,669] INFO Starting ZkClient event thread. >> > (org.I0Itec.zkclient.ZkEventThread) >> > [2013-07-23 20:20:12,679] INFO Client >> > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 >> GMT >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,679] INFO Client environment:host.name=localhost >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,679] INFO Client environment:java.version=1.7.0_21 >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,679] INFO Client environment:java.vendor=Oracle >> > Corporation (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,679] INFO Client >> > environment:java.home=/usr/java/jdk1.7.0_21/jre >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,679] INFO Client >> > >> > >> environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,679] INFO Client >> > >> > >> environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,680] INFO Client environment:java.io.tmpdir=/tmp >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,680] INFO Client environment:java.compiler=<NA> >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,680] INFO Client environment:os.name=Linux >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,680] INFO Client environment:os.arch=amd64 >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,680] INFO Client >> > environment:os.version=2.6.32-279.22.1.el6.x86_64 >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,680] INFO Client environment:user.name=cleartrip >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,680] INFO Client >> environment:user.home=/home/cleartrip >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,680] INFO Client >> > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,681] INFO Initiating client connection, >> connectString= >> > 172.16.18.65:2181 sessionTimeout=30000 >> > watcher=org.I0Itec.zkclient.ZkClient@6926e27 >> > (org.apache.zookeeper.ZooKeeper) >> > [2013-07-23 20:20:12,705] INFO Opening socket connection to server / >> > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) >> > [2013-07-23 20:20:12,710] INFO Socket connection established to zoo1/ >> > 172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn) >> > [2013-07-23 20:20:12,719] INFO Session establishment complete on server >> > zoo1/172.16.18.65:2181, sessionid = 0x13fc72354230160, negotiated >> timeout >> > = >> > 30000 (org.apache.zookeeper.ClientCnxn) >> > [2013-07-23 20:20:12,722] INFO zookeeper state changed (SyncConnected) >> > (org.I0Itec.zkclient.ZkClient) >> > air-DummyProductionConsumerGroup-After,air,1-0 >> > (Group,Topic,BrokerId-PartitionId) >> > Owner = >> > >> > >> air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 >> > Consumer offset = 1140946916 >> > = 1,140,946,916 (1.06G) >> > [2013-07-23 20:20:12,828] ERROR Could not parse broker info >> > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 >> > (kafka.tools.ConsumerOffsetChecker$) >> > air-DummyProductionConsumerGroup-After,air,1-1 >> > (Group,Topic,BrokerId-PartitionId) >> > Owner = >> > >> > >> air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 >> > Consumer offset = 1142891729 >> > = 1,142,891,729 (1.06G) >> > air-DummyProductionConsumerGroup-After,air,2-0 >> > (Group,Topic,BrokerId-PartitionId) >> > Owner = >> > >> > >> air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 >> > Consumer offset = 1587266466 >> > = 1,587,266,466 (1.48G) >> > [2013-07-23 20:20:12,835] ERROR Could not parse broker info >> > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 >> > (kafka.tools.ConsumerOffsetChecker$) >> > air-DummyProductionConsumerGroup-After,air,2-1 >> > (Group,Topic,BrokerId-PartitionId) >> > Owner = >> > >> > >> air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 >> > Consumer offset = 1566027800 >> > = 1,566,027,800 (1.46G) >> > BROKER INFO >> > [2013-07-23 20:20:12,867] INFO Terminate ZkClient event thread. >> > (org.I0Itec.zkclient.ZkEventThread) >> > [2013-07-23 20:20:12,871] INFO EventThread shut down >> > (org.apache.zookeeper.ClientCnxn) >> > [2013-07-23 20:20:12,871] INFO Session: 0x13fc72354230160 closed >> > (org.apache.zookeeper.ZooKeeper) >> > >> > > > > -- > *Samir Madhavan *| Data Scientist | Flutura Business Solutions Pvt. Ltd | > 4th Floor, 'Geetanjali', #693, 15th Cross, J.P Nagar 2nd Phase, > Bangalore, India - 560078 | Mobile: +91 9886139631 | email: * > samir.madha...@fluturasolutions.com*| www.fluturasolutions.com | > -- *Samir Madhavan *| Data Scientist | Flutura Business Solutions Pvt. Ltd | 4th Floor, 'Geetanjali', #693, 15th Cross, J.P Nagar 2nd Phase, Bangalore, India - 560078 | Mobile: +91 9886139631 | email: * samir.madha...@fluturasolutions.com*| www.fluturasolutions.com |
