Re: reading the consumer offsets topic
Hi Tao, Sorry for the delay. Thanks for pointing out that property. That was the fix. On Mon, May 9, 2016 at 6:00 PM, tao xiao wrote: > You need to enable internal topic in the consumer.properties > > exclude.internal.topics=false > > On Mon, 9 May 2016 at 12:42 Cliff Rhyne wrote: > > > Thanks Todd and Tao. I've tried those tricks but no luck. > > > > Just to add more info, this is the __consumer_offsets specific config > that > > is shown by the topic describe command: > > > > > > > Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed > > > > On Mon, May 9, 2016 at 1:16 PM, tao xiao wrote: > > > > > You can try this > > > > > > bin/kafka-console-consumer.sh --consumer.config > > > config/consumer.properties --from-beginning > > > --topic __consumer_offsets --zookeeper localhost:2181 —formatter > > > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" > > > > > > On Mon, 9 May 2016 at 09:40 Todd Palino wrote: > > > > > > > The GroupMetadataManager one should be working for you with 0.9. I > > don’t > > > > have a 0.9 KCC set up at the moment, so I’m using an 0.8 version > where > > > it’s > > > > different (it’s the other class for me). The only thing I can offer > now > > > is > > > > did you put quotes around the arg to --formatter so you don’t get > weird > > > > shell interference? > > > > > > > > -Todd > > > > > > > > > > > > On Mon, May 9, 2016 at 8:18 AM, Cliff Rhyne > wrote: > > > > > > > > > Thanks, Todd. It's still not working unfortunately. > > > > > > > > > > This results in nothing getting printed to the console and requires > > > kill > > > > -9 > > > > > in another window to stop (ctrl-c doesn't work): > > > > > > > > > > /kafka-console-consumer.sh --bootstrap-server localhost:9092 > > > --zookeeper > > > > > --topic __consumer_offsets --formatter > > > > > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > > > > > > > This results in a stack trace because it can't find the class: > > > > > > > > > > ./kafka-console-consumer.sh --bootstrap-server localhost:9092 > > > --zookeeper > > > > > --topic __consumer_offsets --formatter > > > > > kafka.server.OffsetManager\$OffsetsMessageFormatter > > > > > > > > > > Exception in thread "main" java.lang.ClassNotFoundException: > > > > > kafka.server.OffsetManager$OffsetsMessageFormatter > > > > > > > > > > > > > > > I'm on 0.9.0.1. "broker-list" is invalid and zookeeper is required > > > > > regardless of the bootstrap-server parameter. > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Cliff > > > > > > > > > > On Sun, May 8, 2016 at 7:35 PM, Todd Palino > > wrote: > > > > > > > > > > > It looks like you’re just missing the proper message formatter. > Of > > > > > course, > > > > > > that largely depends on your version of the broker. Try: > > > > > > > > > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > > > > > __consumer_offsets > > > > > > --formatter > > > > > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > > > > > > > > > > > > > > > If for some reason that doesn’t work, you can try > > > > > > "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead. > > > > > > > > > > > > -Todd > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne > > > wrote: > > > > > > > > > > > > > I'm having difficulty reading the consumer offsets topic from > the > > > > > command > > > > > > > line. I try the following but it doesn't seem to work (along > > with > > > a > > > > > few > > > > > > >
Re: reading the consumer offsets topic
You need to enable internal topic in the consumer.properties exclude.internal.topics=false On Mon, 9 May 2016 at 12:42 Cliff Rhyne wrote: > Thanks Todd and Tao. I've tried those tricks but no luck. > > Just to add more info, this is the __consumer_offsets specific config that > is shown by the topic describe command: > > > Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed > > On Mon, May 9, 2016 at 1:16 PM, tao xiao wrote: > > > You can try this > > > > bin/kafka-console-consumer.sh --consumer.config > > config/consumer.properties --from-beginning > > --topic __consumer_offsets --zookeeper localhost:2181 —formatter > > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" > > > > On Mon, 9 May 2016 at 09:40 Todd Palino wrote: > > > > > The GroupMetadataManager one should be working for you with 0.9. I > don’t > > > have a 0.9 KCC set up at the moment, so I’m using an 0.8 version where > > it’s > > > different (it’s the other class for me). The only thing I can offer now > > is > > > did you put quotes around the arg to --formatter so you don’t get weird > > > shell interference? > > > > > > -Todd > > > > > > > > > On Mon, May 9, 2016 at 8:18 AM, Cliff Rhyne wrote: > > > > > > > Thanks, Todd. It's still not working unfortunately. > > > > > > > > This results in nothing getting printed to the console and requires > > kill > > > -9 > > > > in another window to stop (ctrl-c doesn't work): > > > > > > > > /kafka-console-consumer.sh --bootstrap-server localhost:9092 > > --zookeeper > > > > --topic __consumer_offsets --formatter > > > > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > > > > > This results in a stack trace because it can't find the class: > > > > > > > > ./kafka-console-consumer.sh --bootstrap-server localhost:9092 > > --zookeeper > > > > --topic __consumer_offsets --formatter > > > > kafka.server.OffsetManager\$OffsetsMessageFormatter > > > > > > > > Exception in thread "main" java.lang.ClassNotFoundException: > > > > kafka.server.OffsetManager$OffsetsMessageFormatter > > > > > > > > > > > > I'm on 0.9.0.1. "broker-list" is invalid and zookeeper is required > > > > regardless of the bootstrap-server parameter. > > > > > > > > > > > > Thanks, > > > > > > > > Cliff > > > > > > > > On Sun, May 8, 2016 at 7:35 PM, Todd Palino > wrote: > > > > > > > > > It looks like you’re just missing the proper message formatter. Of > > > > course, > > > > > that largely depends on your version of the broker. Try: > > > > > > > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > > > > __consumer_offsets > > > > > --formatter > > > > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > > > > > > > > > > > > If for some reason that doesn’t work, you can try > > > > > "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead. > > > > > > > > > > -Todd > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne > > wrote: > > > > > > > > > > > I'm having difficulty reading the consumer offsets topic from the > > > > command > > > > > > line. I try the following but it doesn't seem to work (along > with > > a > > > > few > > > > > > related variants including specifying the zookeeper hosts): > > > > > > > > > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > > > > > __consumer_offsets > > > > > > > > > > > > Is there a special way to read the consumer offsets topic? > > > > > > > > > > > > Thanks, > > > > > > Cliff > > > > > > > > > > > > -- > > > > > > Cliff Rhyne > > > > > > Software Engineering Manager > > > > > > e: crh...@signal.co > > > > > > signal.co > >
Re: reading the consumer offsets topic
Thanks Todd and Tao. I've tried those tricks but no luck. Just to add more info, this is the __consumer_offsets specific config that is shown by the topic describe command: Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed On Mon, May 9, 2016 at 1:16 PM, tao xiao wrote: > You can try this > > bin/kafka-console-consumer.sh --consumer.config > config/consumer.properties --from-beginning > --topic __consumer_offsets --zookeeper localhost:2181 —formatter > "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" > > On Mon, 9 May 2016 at 09:40 Todd Palino wrote: > > > The GroupMetadataManager one should be working for you with 0.9. I don’t > > have a 0.9 KCC set up at the moment, so I’m using an 0.8 version where > it’s > > different (it’s the other class for me). The only thing I can offer now > is > > did you put quotes around the arg to --formatter so you don’t get weird > > shell interference? > > > > -Todd > > > > > > On Mon, May 9, 2016 at 8:18 AM, Cliff Rhyne wrote: > > > > > Thanks, Todd. It's still not working unfortunately. > > > > > > This results in nothing getting printed to the console and requires > kill > > -9 > > > in another window to stop (ctrl-c doesn't work): > > > > > > /kafka-console-consumer.sh --bootstrap-server localhost:9092 > --zookeeper > > > --topic __consumer_offsets --formatter > > > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > > > This results in a stack trace because it can't find the class: > > > > > > ./kafka-console-consumer.sh --bootstrap-server localhost:9092 > --zookeeper > > > --topic __consumer_offsets --formatter > > > kafka.server.OffsetManager\$OffsetsMessageFormatter > > > > > > Exception in thread "main" java.lang.ClassNotFoundException: > > > kafka.server.OffsetManager$OffsetsMessageFormatter > > > > > > > > > I'm on 0.9.0.1. "broker-list" is invalid and zookeeper is required > > > regardless of the bootstrap-server parameter. > > > > > > > > > Thanks, > > > > > > Cliff > > > > > > On Sun, May 8, 2016 at 7:35 PM, Todd Palino wrote: > > > > > > > It looks like you’re just missing the proper message formatter. Of > > > course, > > > > that largely depends on your version of the broker. Try: > > > > > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > > > __consumer_offsets > > > > --formatter > > > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > > > > > > > > > If for some reason that doesn’t work, you can try > > > > "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead. > > > > > > > > -Todd > > > > > > > > > > > > > > > > > > > > On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne > wrote: > > > > > > > > > I'm having difficulty reading the consumer offsets topic from the > > > command > > > > > line. I try the following but it doesn't seem to work (along with > a > > > few > > > > > related variants including specifying the zookeeper hosts): > > > > > > > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > > > > __consumer_offsets > > > > > > > > > > Is there a special way to read the consumer offsets topic? > > > > > > > > > > Thanks, > > > > > Cliff > > > > > > > > > > -- > > > > > Cliff Rhyne > > > > > Software Engineering Manager > > > > > e: crh...@signal.co > > > > > signal.co > > > > > > > > > > > > > > > Cut Through the Noise > > > > > > > > > > This e-mail and any files transmitted with it are for the sole use > of > > > the > > > > > intended recipient(s) and may contain confidential and privileged > > > > > information. Any unauthorized use of this email is strictly > > prohibited. > > > > > ©2016 Signal. All rights reserved. > > > > > > > > > > > > > > > > > > > > > -- > > > > *—-* > > > > *Todd Palino* > > > > Staff Site Reliability Engineer > > > > Data Infrastructure Streaming > > > > > > > > > > > > > > > > linkedin.com/in/toddpalino > > > > > > > > > > > > > > > > -- > > > Cliff Rhyne > > > Software Engineering Manager > > > e: crh...@signal.co > > > signal.co > > > > > > > > > Cut Through the Noise > > > > > > This e-mail and any files transmitted with it are for the sole use of > the > > > intended recipient(s) and may contain confidential and privileged > > > information. Any unauthorized use of this email is strictly prohibited. > > > ©2016 Signal. All rights reserved. > > > > > > > > > > > -- > > *—-* > > *Todd Palino* > > Staff Site Reliability Engineer > > Data Infrastructure Streaming > > > > > > > > linkedin.com/in/toddpalino > > > -- Cliff Rhyne Software Engineering Manager e: crh...@signal.co signal.co Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2016 Signal. All rights reserved.
Re: reading the consumer offsets topic
You can try this bin/kafka-console-consumer.sh --consumer.config config/consumer.properties --from-beginning --topic __consumer_offsets --zookeeper localhost:2181 —formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" On Mon, 9 May 2016 at 09:40 Todd Palino wrote: > The GroupMetadataManager one should be working for you with 0.9. I don’t > have a 0.9 KCC set up at the moment, so I’m using an 0.8 version where it’s > different (it’s the other class for me). The only thing I can offer now is > did you put quotes around the arg to --formatter so you don’t get weird > shell interference? > > -Todd > > > On Mon, May 9, 2016 at 8:18 AM, Cliff Rhyne wrote: > > > Thanks, Todd. It's still not working unfortunately. > > > > This results in nothing getting printed to the console and requires kill > -9 > > in another window to stop (ctrl-c doesn't work): > > > > /kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper > > --topic __consumer_offsets --formatter > > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > This results in a stack trace because it can't find the class: > > > > ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper > > --topic __consumer_offsets --formatter > > kafka.server.OffsetManager\$OffsetsMessageFormatter > > > > Exception in thread "main" java.lang.ClassNotFoundException: > > kafka.server.OffsetManager$OffsetsMessageFormatter > > > > > > I'm on 0.9.0.1. "broker-list" is invalid and zookeeper is required > > regardless of the bootstrap-server parameter. > > > > > > Thanks, > > > > Cliff > > > > On Sun, May 8, 2016 at 7:35 PM, Todd Palino wrote: > > > > > It looks like you’re just missing the proper message formatter. Of > > course, > > > that largely depends on your version of the broker. Try: > > > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > > __consumer_offsets > > > --formatter > > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > > > > > > If for some reason that doesn’t work, you can try > > > "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead. > > > > > > -Todd > > > > > > > > > > > > > > > On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne wrote: > > > > > > > I'm having difficulty reading the consumer offsets topic from the > > command > > > > line. I try the following but it doesn't seem to work (along with a > > few > > > > related variants including specifying the zookeeper hosts): > > > > > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > > > __consumer_offsets > > > > > > > > Is there a special way to read the consumer offsets topic? > > > > > > > > Thanks, > > > > Cliff > > > > > > > > -- > > > > Cliff Rhyne > > > > Software Engineering Manager > > > > e: crh...@signal.co > > > > signal.co > > > > > > > > > > > > Cut Through the Noise > > > > > > > > This e-mail and any files transmitted with it are for the sole use of > > the > > > > intended recipient(s) and may contain confidential and privileged > > > > information. Any unauthorized use of this email is strictly > prohibited. > > > > ©2016 Signal. All rights reserved. > > > > > > > > > > > > > > > > -- > > > *—-* > > > *Todd Palino* > > > Staff Site Reliability Engineer > > > Data Infrastructure Streaming > > > > > > > > > > > > linkedin.com/in/toddpalino > > > > > > > > > > > -- > > Cliff Rhyne > > Software Engineering Manager > > e: crh...@signal.co > > signal.co > > > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2016 Signal. All rights reserved. > > > > > > -- > *—-* > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
Re: reading the consumer offsets topic
The GroupMetadataManager one should be working for you with 0.9. I don’t have a 0.9 KCC set up at the moment, so I’m using an 0.8 version where it’s different (it’s the other class for me). The only thing I can offer now is did you put quotes around the arg to --formatter so you don’t get weird shell interference? -Todd On Mon, May 9, 2016 at 8:18 AM, Cliff Rhyne wrote: > Thanks, Todd. It's still not working unfortunately. > > This results in nothing getting printed to the console and requires kill -9 > in another window to stop (ctrl-c doesn't work): > > /kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper > --topic __consumer_offsets --formatter > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > This results in a stack trace because it can't find the class: > > ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper > --topic __consumer_offsets --formatter > kafka.server.OffsetManager\$OffsetsMessageFormatter > > Exception in thread "main" java.lang.ClassNotFoundException: > kafka.server.OffsetManager$OffsetsMessageFormatter > > > I'm on 0.9.0.1. "broker-list" is invalid and zookeeper is required > regardless of the bootstrap-server parameter. > > > Thanks, > > Cliff > > On Sun, May 8, 2016 at 7:35 PM, Todd Palino wrote: > > > It looks like you’re just missing the proper message formatter. Of > course, > > that largely depends on your version of the broker. Try: > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > __consumer_offsets > > --formatter > kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > > > > If for some reason that doesn’t work, you can try > > "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead. > > > > -Todd > > > > > > > > > > On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne wrote: > > > > > I'm having difficulty reading the consumer offsets topic from the > command > > > line. I try the following but it doesn't seem to work (along with a > few > > > related variants including specifying the zookeeper hosts): > > > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > > __consumer_offsets > > > > > > Is there a special way to read the consumer offsets topic? > > > > > > Thanks, > > > Cliff > > > > > > -- > > > Cliff Rhyne > > > Software Engineering Manager > > > e: crh...@signal.co > > > signal.co > > > > > > > > > Cut Through the Noise > > > > > > This e-mail and any files transmitted with it are for the sole use of > the > > > intended recipient(s) and may contain confidential and privileged > > > information. Any unauthorized use of this email is strictly prohibited. > > > ©2016 Signal. All rights reserved. > > > > > > > > > > > -- > > *—-* > > *Todd Palino* > > Staff Site Reliability Engineer > > Data Infrastructure Streaming > > > > > > > > linkedin.com/in/toddpalino > > > > > > -- > Cliff Rhyne > Software Engineering Manager > e: crh...@signal.co > signal.co > > > Cut Through the Noise > > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. Any unauthorized use of this email is strictly prohibited. > ©2016 Signal. All rights reserved. > -- *—-* *Todd Palino* Staff Site Reliability Engineer Data Infrastructure Streaming linkedin.com/in/toddpalino
Re: reading the consumer offsets topic
Thanks, Todd. It's still not working unfortunately. This results in nothing getting printed to the console and requires kill -9 in another window to stop (ctrl-c doesn't work): /kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper --topic __consumer_offsets --formatter kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter This results in a stack trace because it can't find the class: ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper --topic __consumer_offsets --formatter kafka.server.OffsetManager\$OffsetsMessageFormatter Exception in thread "main" java.lang.ClassNotFoundException: kafka.server.OffsetManager$OffsetsMessageFormatter I'm on 0.9.0.1. "broker-list" is invalid and zookeeper is required regardless of the bootstrap-server parameter. Thanks, Cliff On Sun, May 8, 2016 at 7:35 PM, Todd Palino wrote: > It looks like you’re just missing the proper message formatter. Of course, > that largely depends on your version of the broker. Try: > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > __consumer_offsets > --formatter kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter > > > If for some reason that doesn’t work, you can try > "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead. > > -Todd > > > > > On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne wrote: > > > I'm having difficulty reading the consumer offsets topic from the command > > line. I try the following but it doesn't seem to work (along with a few > > related variants including specifying the zookeeper hosts): > > > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > > __consumer_offsets > > > > Is there a special way to read the consumer offsets topic? > > > > Thanks, > > Cliff > > > > -- > > Cliff Rhyne > > Software Engineering Manager > > e: crh...@signal.co > > signal.co > > > > > > Cut Through the Noise > > > > This e-mail and any files transmitted with it are for the sole use of the > > intended recipient(s) and may contain confidential and privileged > > information. Any unauthorized use of this email is strictly prohibited. > > ©2016 Signal. All rights reserved. > > > > > > -- > *—-* > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino > -- Cliff Rhyne Software Engineering Manager e: crh...@signal.co signal.co Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2016 Signal. All rights reserved.
Re: reading the consumer offsets topic
It looks like you’re just missing the proper message formatter. Of course, that largely depends on your version of the broker. Try: ./kafka-console-consumer.sh --broker-list localhost:9092 --topic __consumer_offsets --formatter kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter If for some reason that doesn’t work, you can try "kafka.server.OffsetManager\$OffsetsMessageFormatter” instead. -Todd On Sun, May 8, 2016 at 1:26 PM, Cliff Rhyne wrote: > I'm having difficulty reading the consumer offsets topic from the command > line. I try the following but it doesn't seem to work (along with a few > related variants including specifying the zookeeper hosts): > > ./kafka-console-consumer.sh --broker-list localhost:9092 --topic > __consumer_offsets > > Is there a special way to read the consumer offsets topic? > > Thanks, > Cliff > > -- > Cliff Rhyne > Software Engineering Manager > e: crh...@signal.co > signal.co > > > Cut Through the Noise > > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. Any unauthorized use of this email is strictly prohibited. > ©2016 Signal. All rights reserved. > -- *—-* *Todd Palino* Staff Site Reliability Engineer Data Infrastructure Streaming linkedin.com/in/toddpalino
reading the consumer offsets topic
I'm having difficulty reading the consumer offsets topic from the command line. I try the following but it doesn't seem to work (along with a few related variants including specifying the zookeeper hosts): ./kafka-console-consumer.sh --broker-list localhost:9092 --topic __consumer_offsets Is there a special way to read the consumer offsets topic? Thanks, Cliff -- Cliff Rhyne Software Engineering Manager e: crh...@signal.co signal.co Cut Through the Noise This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use of this email is strictly prohibited. ©2016 Signal. All rights reserved.