Not sure about the error. However, your consumer seems to be lagging. Are the consumer offsets moving at all? If not, are your consumer threads still alive? You probably want to try/catch the consumer code to see if there is any unexpected exceptions.
Thanks, Jun On Tue, Jul 23, 2013 at 7:58 AM, Anurup Raveendran < anurup.raveend...@fluturasolutions.com> wrote: > After consuming about 5 gb of messages , it stopped consuming and it got > stuck at a particular offset. After running the following command I got an > error regarding the broker. > > > > > $ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181--topic > air > > > > [2013-07-23 20:19:42,748] INFO Starting ZkClient event thread. > (org.I0Itec.zkclient.ZkEventThread) > [2013-07-23 20:19:42,757] INFO Client > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,757] INFO Client environment:host.name=localhost > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,757] INFO Client environment:java.version=1.7.0_21 > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,757] INFO Client environment:java.vendor=Oracle > Corporation (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,757] INFO Client > environment:java.home=/usr/java/jdk1.7.0_21/jre > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,757] INFO Client > > environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client > > environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client environment:java.io.tmpdir=/tmp > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client environment:java.compiler=<NA> > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client environment:os.name=Linux > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client environment:os.arch=amd64 > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client > environment:os.version=2.6.32-279.22.1.el6.x86_64 > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client environment:user.name=cleartrip > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client environment:user.home=/home/cleartrip > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,758] INFO Client > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,759] INFO Initiating client connection, connectString= > 172.16.18.65:2181 sessionTimeout=30000 > watcher=org.I0Itec.zkclient.ZkClient@6926e27 > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:19:42,783] INFO Opening socket connection to server / > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) > [2013-07-23 20:19:42,788] INFO Socket connection established to zoo1/ > 172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn) > [2013-07-23 20:19:42,800] INFO Session establishment complete on server > zoo1/172.16.18.65:2181, sessionid = 0x13fc7235423015f, negotiated timeout > = > 30000 (org.apache.zookeeper.ClientCnxn) > [2013-07-23 20:19:42,802] INFO zookeeper state changed (SyncConnected) > (org.I0Itec.zkclient.ZkClient) > air-DummyProductionConsumerGroup-After,air,1-0 > (Group,Topic,BrokerId-PartitionId) > Owner = > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 > Consumer offset = 1140946916 > = 1,140,946,916 (1.06G) > [2013-07-23 20:19:42,948] ERROR Could not parse broker info > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 > (kafka.tools.ConsumerOffsetChecker$) > air-DummyProductionConsumerGroup-After,air,1-1 > (Group,Topic,BrokerId-PartitionId) > Owner = > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 > Consumer offset = 1142891729 > = 1,142,891,729 (1.06G) > air-DummyProductionConsumerGroup-After,air,2-0 > (Group,Topic,BrokerId-PartitionId) > Owner = > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 > Consumer offset = 1587266466 > = 1,587,266,466 (1.48G) > [2013-07-23 20:19:42,956] ERROR Could not parse broker info > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 > (kafka.tools.ConsumerOffsetChecker$) > air-DummyProductionConsumerGroup-After,air,2-1 > (Group,Topic,BrokerId-PartitionId) > Owner = > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 > Consumer offset = 1566027800 > = 1,566,027,800 (1.46G) > BROKER INFO > [2013-07-23 20:19:42,988] INFO Terminate ZkClient event thread. > (org.I0Itec.zkclient.ZkEventThread) > [2013-07-23 20:19:42,992] INFO EventThread shut down > (org.apache.zookeeper.ClientCnxn) > [2013-07-23 20:19:42,992] INFO Session: 0x13fc7235423015f closed > (org.apache.zookeeper.ZooKeeper) > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ clear > > [cleartrip@d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group > air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181--topic > air > [2013-07-23 20:20:12,669] INFO Starting ZkClient event thread. > (org.I0Itec.zkclient.ZkEventThread) > [2013-07-23 20:20:12,679] INFO Client > environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,679] INFO Client environment:host.name=localhost > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,679] INFO Client environment:java.version=1.7.0_21 > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,679] INFO Client environment:java.vendor=Oracle > Corporation (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,679] INFO Client > environment:java.home=/usr/java/jdk1.7.0_21/jre > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,679] INFO Client > > environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,679] INFO Client > > environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,680] INFO Client environment:java.io.tmpdir=/tmp > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,680] INFO Client environment:java.compiler=<NA> > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,680] INFO Client environment:os.name=Linux > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,680] INFO Client environment:os.arch=amd64 > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,680] INFO Client > environment:os.version=2.6.32-279.22.1.el6.x86_64 > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,680] INFO Client environment:user.name=cleartrip > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,680] INFO Client environment:user.home=/home/cleartrip > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,680] INFO Client > environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,681] INFO Initiating client connection, connectString= > 172.16.18.65:2181 sessionTimeout=30000 > watcher=org.I0Itec.zkclient.ZkClient@6926e27 > (org.apache.zookeeper.ZooKeeper) > [2013-07-23 20:20:12,705] INFO Opening socket connection to server / > 172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn) > [2013-07-23 20:20:12,710] INFO Socket connection established to zoo1/ > 172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn) > [2013-07-23 20:20:12,719] INFO Session establishment complete on server > zoo1/172.16.18.65:2181, sessionid = 0x13fc72354230160, negotiated timeout > = > 30000 (org.apache.zookeeper.ClientCnxn) > [2013-07-23 20:20:12,722] INFO zookeeper state changed (SyncConnected) > (org.I0Itec.zkclient.ZkClient) > air-DummyProductionConsumerGroup-After,air,1-0 > (Group,Topic,BrokerId-PartitionId) > Owner = > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0 > Consumer offset = 1140946916 > = 1,140,946,916 (1.06G) > [2013-07-23 20:20:12,828] ERROR Could not parse broker info > d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092 > (kafka.tools.ConsumerOffsetChecker$) > air-DummyProductionConsumerGroup-After,air,1-1 > (Group,Topic,BrokerId-PartitionId) > Owner = > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1 > Consumer offset = 1142891729 > = 1,142,891,729 (1.06G) > air-DummyProductionConsumerGroup-After,air,2-0 > (Group,Topic,BrokerId-PartitionId) > Owner = > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0 > Consumer offset = 1587266466 > = 1,587,266,466 (1.48G) > [2013-07-23 20:20:12,835] ERROR Could not parse broker info > d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092 > (kafka.tools.ConsumerOffsetChecker$) > air-DummyProductionConsumerGroup-After,air,2-1 > (Group,Topic,BrokerId-PartitionId) > Owner = > > air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1 > Consumer offset = 1566027800 > = 1,566,027,800 (1.46G) > BROKER INFO > [2013-07-23 20:20:12,867] INFO Terminate ZkClient event thread. > (org.I0Itec.zkclient.ZkEventThread) > [2013-07-23 20:20:12,871] INFO EventThread shut down > (org.apache.zookeeper.ClientCnxn) > [2013-07-23 20:20:12,871] INFO Session: 0x13fc72354230160 closed > (org.apache.zookeeper.ZooKeeper) >
