Guozhang, Currently we are in the stage to testing producer, our C# producer sending data to brokers, and use
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance command to produce the messages. We don't have a coded consumer to commit offset, we use bin/kafka-console-consumer.sh --zookeeper command to consume, is there a command that we can use on command line to create zk path? thanks AL On Mon, Jan 19, 2015 at 4:14 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Sa, > > Did your consumer ever commit offsets to Kafka? If not then no > corresponding ZK path will be created. > > Guozhang > > On Mon, Jan 19, 2015 at 3:58 PM, Sa Li <sal...@gmail.com> wrote: > > > Hi, > > > > I use such tool > > > > Consumer Offset Checker > > > > Displays the: Consumer Group, Topic, Partitions, Offset, logSize, Lag, > > Owner for the specified set of Topics and Consumer Group > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker > > > > To be able to know the consumer group, in zkCli.sh > > > > [zk: localhost:2181(CONNECTED) 3] ls / > > [transactional, admin, zookeeper, consumers, config, controller, storm, > > brokers, controller_epoch] > > [zk: localhost:2181(CONNECTED) 4] ls /consumers > > [web-console-consumer-99295, web-console-consumer-37853, > > web-console-consumer-30841, perf-consumer-92283, perf-consumer-21631, > > perf-consumer-95281, perf-consumer-59296, web-console-consumer-52126, > > web-console-consumer-89137, perf-consumer-72484, perf-consumer-80363, > > web-console-consumer-47543, web-console-consumer-22509, > > perf-consumer-16954, perf-consumer-53957, perf-consumer-39448, > > web-console-consumer-17021, perf-consumer-88693, > > web-console-consumer-48744, web-console-consumer-82543, > > perf-consumer-89565, web-console-consumer-97959, perf-consumer-40427, > > web-console-consumer-95350, web-console-consumer-26473, > > web-console-consumer-79384, web-console-consumer-87777, > > perf-consumer-91681, web-console-consumer-36136, > > web-console-consumer-86924, perf-consumer-24510, perf-consumer-5888, > > perf-consumer-73534, perf-consumer-92985, perf-consumer-7675, > > perf-consumer-52306, perf-consumer-87352, web-console-consumer-30400] > > [zk: localhost:2181(CONNECTED) 5] > > > > I then run > > > > root@exemplary-birds:/srv/kafka# bin/kafka-run-class.sh > > kafka.tools.ConsumerOffsetChecker --topic PofApiTest-1 --group > > web-console-consumer-48744 > > Group Topic Pid Offset > > logSize Lag Owner > > Exception in thread "main" > org.I0Itec.zkclient.exception.ZkNoNodeException: > > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > > NoNode for /consumers/web-console-consumer-48744/offsets/PofApiTest-1/0 > > at > > org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47) > > at > > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685) > > at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:766) > > at org.I0Itec.zkclient.ZkClient.readData(ZkClient.java:761) > > at kafka.utils.ZkUtils$.readData(ZkUtils.scala:461) > > at > > > > > kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processPartition(ConsumerOffsetChecker.scala:59) > > at > > > > > kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply$mcVI$sp(ConsumerOffsetChecker.scala:89) > > at > > > > > kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:89) > > at > > > > > kafka.tools.ConsumerOffsetChecker$$anonfun$kafka$tools$ConsumerOffsetChecker$$processTopic$1.apply(ConsumerOffsetChecker.scala:89) > > at > > > > > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > > at > > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) > > at > > > > > kafka.tools.ConsumerOffsetChecker$.kafka$tools$ConsumerOffsetChecker$$processTopic(ConsumerOffsetChecker.scala:88) > > at > > > > > kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:153) > > at > > > > > kafka.tools.ConsumerOffsetChecker$$anonfun$main$3.apply(ConsumerOffsetChecker.scala:153) > > at scala.collection.immutable.List.foreach(List.scala:318) > > at > > kafka.tools.ConsumerOffsetChecker$.main(ConsumerOffsetChecker.scala:152) > > at > > kafka.tools.ConsumerOffsetChecker.main(ConsumerOffsetChecker.scala) > > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > > KeeperErrorCode = NoNode for > > /consumers/web-console-consumer-48744/offsets/PofApiTest-1/0 > > at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:102) > > at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:956) > > at > org.I0Itec.zkclient.ZkConnection.readData(ZkConnection.java:103) > > at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:770) > > at org.I0Itec.zkclient.ZkClient$9.call(ZkClient.java:766) > > at > > org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) > > ... 15 more > > > > So consumer groups make confusing, I didn't specify the consumer-group-id > > in producer, the only place I know to config group is consumer.properties > > > > > > #consumer group id > > group.id=test-consumer-group > > > > > > Any hints? Thanks > > > > > > AL > > > > > > > > > > > > > > On Mon, Dec 15, 2014 at 6:46 PM, nitin sharma < > kumarsharma.ni...@gmail.com > > > > > wrote: > > > > > got it ... thanks a lot. > > > > > > Regards, > > > Nitin Kumar Sharma. > > > > > > > > > On Mon, Dec 15, 2014 at 9:26 PM, Gwen Shapira <gshap...@cloudera.com> > > > wrote: > > > > > > > > Hi Nitin, > > > > > > > > Go to where you installed zookeeper and run: > > > > > > > > bin/zkCli.sh -server 127.0.0.1:2181 > > > > > > > > On Mon, Dec 15, 2014 at 6:09 PM, nitin sharma > > > > <kumarsharma.ni...@gmail.com> wrote: > > > > > Thanks Neha and Gwen for your responses.. > > > > > > > > > > @Gwen -- Kindly explain how to perform the steps you have > mentioned. > > > how > > > > > should i connect to a zookeeper..? > > > > > > > > > > Regards, > > > > > Nitin Kumar Sharma. > > > > > > > > > > > > > > > On Mon, Dec 15, 2014 at 6:36 PM, Neha Narkhede <n...@confluent.io> > > > > wrote: > > > > >> > > > > >> In addition to Gwen's suggestion, we actually don't have jmx > metrics > > > > that > > > > >> give you a list of actively consuming processes. > > > > >> > > > > >> On Mon, Dec 15, 2014 at 12:59 PM, Gwen Shapira < > > gshap...@cloudera.com > > > > > > > > >> wrote: > > > > >> > > > > > >> > Currently you can find the number of consumer groups through > > > > ZooKeeper: > > > > >> > > > > > >> > connect to ZK and run > > > > >> > ls /consumers > > > > >> > > > > > >> > and count the number of results > > > > >> > > > > > >> > On Mon, Dec 15, 2014 at 11:34 AM, nitin sharma > > > > >> > <kumarsharma.ni...@gmail.com> wrote: > > > > >> > > Hi Team, > > > > >> > > > > > > >> > > Is it possible to know how many Consumer Group connected to > > kafka > > > > >> broker > > > > >> > Ids > > > > >> > > and as well as how many Instances within a Group are fetching > > > > messages > > > > >> > from > > > > >> > > Kafka Brokers > > > > >> > > > > > > >> > > Regards, > > > > >> > > Nitin Kumar Sharma. > > > > >> > > > > > >> > > > > >> > > > > >> -- > > > > >> Thanks, > > > > >> Neha > > > > >> > > > > > > > > > > > > > > > -- > > > > Alec Li > > > > > > -- > -- Guozhang > -- Alec Li
