Hi Raja, Thanks a lot for that input. It definitely was a problem with the __consumer_offsets not getting updated (not sure why that didnt happen though with upgrading kafka with a new release). But I deleted the __consumer_offsets topic and it was auto created and the consumer offset checker works fine now.
Thanks, Meghana On Thu, May 14, 2015 at 11:00 AM, Rajasekar Elango <rela...@salesforce.com> wrote: > Hi Meghana, > > We also faced similar issue and found that it returned > ConsumerCoordinatorNotAvailableCode always for one broker (server id 3) and > leader for all partitions of __consumer_offsets topic is same broker id 3. > > So wiped off kafka data dir on that broker and restarted it. After that > ConsumerOffsetChecker started working. We had replication enabled so losing > kafka data dir is not big deal for us. If you can't delete complete kafka > data dir, you can try deleting just __consumer_offsets data. > > Thanks, > Raja. > > > On Thu, May 14, 2015 at 10:46 AM, Meghana Narasimhan < > mnarasim...@bandwidth.com> wrote: > > > Hi Mayuresh, > > A few more inputs that I can provide at the moment after some testing are > > as follows. > > 1. The error returned by the consumer offset checker's > > ConsumerMetadataResponse is "ConsumerCoordinatorNotAvailableCode". Could > it > > somehow be related to the offsets being written to zookeeper and not the > > internal kafka offsets topic ? > > 2. I see that the __consumer_offsets topic has been created on all the > > brokers. > > 3. I also tried the steps for migrating from zookeeper to kafka offset > > topic as specified in the documentation [using the offsets.storage and > > dual.commit.enabled configs] > > 4. Also based on a few other links I tried to commit offset > > using OffsetCommitRequest method and to get ConsumerMetadataResponse . > But > > the OffsetCommitResponse also returned error and could not commit the > > offset successfully. > > > > On the other hand the producer and consumer are working fine and able to > > produce and consume data from the topic. The issue is only with the > > consumer offset checker tool. > > > > Thanks, > > Meghana > > > > > > On Mon, May 11, 2015 at 7:34 PM, Mayuresh Gharat < > > gharatmayures...@gmail.com > > > wrote: > > > > > Hi Meghana, > > > > > > Let me try this out on my cluster that has latest trunk deployed. > > > > > > Thanks, > > > > > > Mayuresh > > > > > > On Mon, May 11, 2015 at 1:53 PM, Meghana Narasimhan < > > > mnarasim...@bandwidth.com> wrote: > > > > > > > Hi Mayuresh, > > > > A small update. The Kafka version I'm currently using is > 2.10-0.8.2.1 > > > (not > > > > 2.11 as previously mentioned). The cluster looks fine. Not sure why > the > > > > consumer offset checker does not return a valid output and gets > stuck. > > > > > > > > bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic > test > > > > Topic:test PartitionCount:3 ReplicationFactor:3 > > > > Configs:min.insync.replicas=2 > > > > Topic: test Partition: 0 Leader: 1 Replicas: 1,2,0 > > Isr: > > > > 1,2,0 > > > > Topic: test Partition: 1 Leader: 2 Replicas: 2,0,1 > > Isr: > > > > 1,2,0 > > > > Topic: test Partition: 2 Leader: 0 Replicas: 0,1,2 > > Isr: > > > > 1,2,0 > > > > > > > > > > > > > > > > > > > > On Fri, May 8, 2015 at 12:52 PM, Meghana Narasimhan < > > > > mnarasim...@bandwidth.com> wrote: > > > > > > > > > Hi Mayuresh, > > > > > > > > > > Yes, the broker is up and accepting connections. Multiple consumers > > are > > > > > consuming off topics on the broker. > > > > > Also I am seeing the issue only with this particular version ( > > > > > 2.11-0.8.2.1). It worked fine with the beta that I was using > earlier. > > > > > > > > > > > > > > > On Fri, May 8, 2015 at 12:45 PM, Mayuresh Gharat < > > > > > gharatmayures...@gmail.com> wrote: > > > > > > > > > >> Is X.X.X.X:9092 up and accepting connections? > > > > >> I am confused aas in why is it not connecting some other broker if > > > > >> connection to this broker fails. Can you check if the broker is > up? > > > > >> > > > > >> The way it works is the consumer will send a > ConsumerMetadataRequest > > > to > > > > >> one > > > > >> of the brokers and get the offsetmanager for its group and then > > > perform > > > > >> the > > > > >> offset management. > > > > >> > > > > >> Thanks, > > > > >> > > > > >> Mayuresh > > > > >> > > > > >> On Fri, May 8, 2015 at 9:22 AM, Meghana Narasimhan < > > > > >> mnarasim...@bandwidth.com> wrote: > > > > >> > > > > >> > Hi, > > > > >> > I'm using the Kafka 8.2.1 version(kafka_2.11-0.8.2.1) and the > > > consumer > > > > >> > offset checker hangs indefinitely and does not return any > > results. I > > > > >> > enabled the debug for tools and below is the debug statements as > > > seen > > > > on > > > > >> > the stdout. Any thoughts or inputs on this will be much > > appreciated. > > > > >> > > > > > >> > command used : > > > > >> > bin/kafka-consumer-offset-checker.sh --zookeeper localhost:2181 > > > > --group > > > > >> > test-consumer-group > > > > >> > or > > > > >> > ./kafka-consumer-offset-checker.sh --zookeeper > > > > >> > broker1:2181,broker2:2181,broker3:2181 --group > test-consumer-group > > > > >> > > > > > >> > DEBUG Querying X.X.X.X:9092 to locate offset manager for > > > > >> > test-consumer-group. (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:23:55,090] DEBUG Consumer metadata response: > > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:23:55,091] DEBUG Query to X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > > >> milliseconds. > > > > >> > (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:23:58,093] DEBUG Querying X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:23:58,102] DEBUG Consumer metadata response: > > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:23:58,103] DEBUG Query to X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > > >> milliseconds. > > > > >> > (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:01,107] DEBUG Querying X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:01,115] DEBUG Consumer metadata response: > > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:01,116] DEBUG Query to X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > > >> milliseconds. > > > > >> > (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:04,119] DEBUG Querying X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:04,124] DEBUG Consumer metadata response: > > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:04,126] DEBUG Query to X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > > >> milliseconds. > > > > >> > (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:04,993] DEBUG Got ping response for sessionid: > > > > >> > 0x14d33e7fbc80002 after 3ms (org.apache.zookeeper.ClientCnxn) > > > > >> > [2015-05-08 10:24:07,127] DEBUG Querying X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:07,131] DEBUG Consumer metadata response: > > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:07,132] DEBUG Query to X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > > >> milliseconds. > > > > >> > (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:10,132] DEBUG Querying X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:10,138] DEBUG Consumer metadata response: > > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:10,139] DEBUG Query to X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > > >> milliseconds. > > > > >> > (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:13,143] DEBUG Querying X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group. (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:13,151] DEBUG Consumer metadata response: > > > > >> > ConsumerMetadataResponse(None,15,0) (kafka.client.ClientUtils$) > > > > >> > [2015-05-08 10:24:13,152] DEBUG Query to X.X.X.X:9092 to locate > > > offset > > > > >> > manager for test-consumer-group failed - will retry in 3000 > > > > >> milliseconds. > > > > >> > (kafka.client.ClientUtils$) > > > > >> > > > > > >> > Thanks, > > > > >> > Meghana > > > > >> > > > > > >> > > > > >> > > > > >> > > > > > > > > > > > > -- > Thanks, > Raja. >
