One of the common pattern we observed when we ran consumer from start multiple times is that it catches up till the data that producer has produced and then it just gets hung. If we kill it and then start it again, it starts consuming and then again it just gets hung. We have added the try catch statements but there seems to be no error coming.
On Tue, Jul 23, 2013 at 9:06 PM, Jun Rao <jun...@gmail.com> wrote: > Not sure about the error. However, your consumer seems to be lagging. Are > the consumer offsets moving at all? If not, are your consumer threads still > alive? You probably want to try/catch the consumer code to see if there is > any unexpected exceptions. > > Thanks, > > Jun > > > On Tue, Jul 23, 2013 at 7:58 AM, Anurup Raveendran < > anurup.raveend...@fluturasolutions.com> wrote: > > > After consuming about 5 gb of messages , it stopped consuming and it got > > stuck at a particular offset. After running the following command I got > an > > error regarding the broker. > > > > > > > > > > $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181 > --topic > > air > > > > > > > > [2013-07-23 20:19:42,748] INFO Starting ZkClient event thread. > > (org.I0Itec.zkclient.ZkEventThread) > > [2013-07-23 20:19:42,757] INFO Client > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 > GMT > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,757] INFO Client environment:host.name=localhost > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,757] INFO Client environment:java.version=1.7.0_21 > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,757] INFO Client environment:java.vendor=Oracle > > Corporation (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,757] INFO Client > > environment:java.home=/usr/java/jdk1.7.0_21/jre > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,757] INFO Client > > > > > environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client > > > > > environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client environment:java.io.tmpdir=/tmp > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client environment:java.compiler=<NA> > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client environment:os.name=Linux > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client environment:os.arch=amd64 > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client > > environment:os.version=2.6.32-279.22.1.el6.x86_64 > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client environment:user.name=cleartrip > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client > environment:user.home=/home/cleartrip > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,758] INFO Client > > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,759] INFO Initiating client connection, > connectString= > > 172.16.18.65:2181 sessionTimeout=30000 > > watcher=org.I0Itec.zkclient.ZkClient@6926e27 > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:19:42,783] INFO Opening socket connection to server / > > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) > > [2013-07-23 20:19:42,788] INFO Socket connection established to zoo1/ > > 172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn) > > [2013-07-23 20:19:42,800] INFO Session establishment complete on server > > zoo1/172.16.18.65:2181, sessionid = 0x13fc7235423015f, negotiated > timeout > > = > > 30000 (org.apache.zookeeper.ClientCnxn) > > [2013-07-23 20:19:42,802] INFO zookeeper state changed (SyncConnected) > > (org.I0Itec.zkclient.ZkClient) > > air-DummyProductionConsumerGroup-After,air,1-0 > > (Group,Topic,BrokerId-PartitionId) > > Owner = > > > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 > > Consumer offset = 1140946916 > > = 1,140,946,916 (1.06G) > > [2013-07-23 20:19:42,948] ERROR Could not parse broker info > > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 > > (kafka.tools.ConsumerOffsetChecker$) > > air-DummyProductionConsumerGroup-After,air,1-1 > > (Group,Topic,BrokerId-PartitionId) > > Owner = > > > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 > > Consumer offset = 1142891729 > > = 1,142,891,729 (1.06G) > > air-DummyProductionConsumerGroup-After,air,2-0 > > (Group,Topic,BrokerId-PartitionId) > > Owner = > > > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 > > Consumer offset = 1587266466 > > = 1,587,266,466 (1.48G) > > [2013-07-23 20:19:42,956] ERROR Could not parse broker info > > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 > > (kafka.tools.ConsumerOffsetChecker$) > > air-DummyProductionConsumerGroup-After,air,2-1 > > (Group,Topic,BrokerId-PartitionId) > > Owner = > > > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 > > Consumer offset = 1566027800 > > = 1,566,027,800 (1.46G) > > BROKER INFO > > [2013-07-23 20:19:42,988] INFO Terminate ZkClient event thread. > > (org.I0Itec.zkclient.ZkEventThread) > > [2013-07-23 20:19:42,992] INFO EventThread shut down > > (org.apache.zookeeper.ClientCnxn) > > [2013-07-23 20:19:42,992] INFO Session: 0x13fc7235423015f closed > > (org.apache.zookeeper.ZooKeeper) > > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ clear > > > > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181 > --topic > > air > > [2013-07-23 20:20:12,669] INFO Starting ZkClient event thread. > > (org.I0Itec.zkclient.ZkEventThread) > > [2013-07-23 20:20:12,679] INFO Client > > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 > GMT > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,679] INFO Client environment:host.name=localhost > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,679] INFO Client environment:java.version=1.7.0_21 > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,679] INFO Client environment:java.vendor=Oracle > > Corporation (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,679] INFO Client > > environment:java.home=/usr/java/jdk1.7.0_21/jre > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,679] INFO Client > > > > > environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,679] INFO Client > > > > > environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,680] INFO Client environment:java.io.tmpdir=/tmp > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,680] INFO Client environment:java.compiler=<NA> > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,680] INFO Client environment:os.name=Linux > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,680] INFO Client environment:os.arch=amd64 > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,680] INFO Client > > environment:os.version=2.6.32-279.22.1.el6.x86_64 > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,680] INFO Client environment:user.name=cleartrip > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,680] INFO Client > environment:user.home=/home/cleartrip > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,680] INFO Client > > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,681] INFO Initiating client connection, > connectString= > > 172.16.18.65:2181 sessionTimeout=30000 > > watcher=org.I0Itec.zkclient.ZkClient@6926e27 > > (org.apache.zookeeper.ZooKeeper) > > [2013-07-23 20:20:12,705] INFO Opening socket connection to server / > > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) > > [2013-07-23 20:20:12,710] INFO Socket connection established to zoo1/ > > 172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn) > > [2013-07-23 20:20:12,719] INFO Session establishment complete on server > > zoo1/172.16.18.65:2181, sessionid = 0x13fc72354230160, negotiated > timeout > > = > > 30000 (org.apache.zookeeper.ClientCnxn) > > [2013-07-23 20:20:12,722] INFO zookeeper state changed (SyncConnected) > > (org.I0Itec.zkclient.ZkClient) > > air-DummyProductionConsumerGroup-After,air,1-0 > > (Group,Topic,BrokerId-PartitionId) > > Owner = > > > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 > > Consumer offset = 1140946916 > > = 1,140,946,916 (1.06G) > > [2013-07-23 20:20:12,828] ERROR Could not parse broker info > > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 > > (kafka.tools.ConsumerOffsetChecker$) > > air-DummyProductionConsumerGroup-After,air,1-1 > > (Group,Topic,BrokerId-PartitionId) > > Owner = > > > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 > > Consumer offset = 1142891729 > > = 1,142,891,729 (1.06G) > > air-DummyProductionConsumerGroup-After,air,2-0 > > (Group,Topic,BrokerId-PartitionId) > > Owner = > > > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 > > Consumer offset = 1587266466 > > = 1,587,266,466 (1.48G) > > [2013-07-23 20:20:12,835] ERROR Could not parse broker info > > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 > > (kafka.tools.ConsumerOffsetChecker$) > > air-DummyProductionConsumerGroup-After,air,2-1 > > (Group,Topic,BrokerId-PartitionId) > > Owner = > > > > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 > > Consumer offset = 1566027800 > > = 1,566,027,800 (1.46G) > > BROKER INFO > > [2013-07-23 20:20:12,867] INFO Terminate ZkClient event thread. > > (org.I0Itec.zkclient.ZkEventThread) > > [2013-07-23 20:20:12,871] INFO EventThread shut down > > (org.apache.zookeeper.ClientCnxn) > > [2013-07-23 20:20:12,871] INFO Session: 0x13fc72354230160 closed > > (org.apache.zookeeper.ZooKeeper) > > > -- *Samir Madhavan *| Data Scientist | Flutura Business Solutions Pvt. Ltd | 4th Floor, 'Geetanjali', #693, 15th Cross, J.P Nagar 2nd Phase, Bangalore, India - 560078 | Mobile: +91 9886139631 | email: * samir.madha...@fluturasolutions.com*| www.fluturasolutions.com |
