Matthias, I will try to get you the broker logs. As far as the partitions of __consumer_offsets topic, I am not quite clear on what you want me to provide. Are you wanting the output of
./kafka-topics.sh --zookeeper zk-host --describe --topic __consumer_offsets -russ On Fri, Feb 16, 2018 at 1:45 PM, Matthias J. Sax <matth...@confluent.io> wrote: > Ok. Let us know if bouncing works. > > Sounds like a bug thought, that the consumer group is not cleared. It > would be helpful to get the broker logs (of the node that hosts the > coordinator). Can you also provide a dump of the corresponding > partitions of __consumer_offsets topic? > > Thanks a lot. > > -Matthias > > > On 2/16/18 12:35 PM, Russell Teabeault wrote: > > Matthias, > > > > The session.timeout.ms is set to 10000. It has been in this "weird" > state > > now for going on 24 hours. > > We are going to try to bounce the group coordinator. > > > > Thanks, > > -russ > > > > On Fri, Feb 16, 2018 at 12:18 PM, Matthias J. Sax <matth...@confluent.io > > > > wrote: > > > >> Hi, > >> > >> this "weird" state can happen. Usually, a consumer sends a "leave group > >> request" to inform the broker it's shutting down. However, this is a > >> send-and-forget approach and if the message is lost, the broker might > >> think that a stopped consumer is still alive. For this reason, the > >> broker should eventually time out consumers and thus clean-up the > >> consumer group. > >> > >> How long did you wait? > >> > >> Did you modify the `session.timeout.ms` config? > >> > >> As a last resort, bouncing the broker that hosts the consumer group > >> coordinator should resolve the issue. > >> > >> -Matthias > >> > >> On 2/15/18 10:09 PM, Russell Teabeault wrote: > >>> It appears that kafka and/or kafka streams has gotten itself into a > weird > >>> state. My KS app is not running. I try to reset it using the reset > >> command > >>> line tool and I get: > >>> > >>> ERROR: java.lang.IllegalStateException: Consumer group > >>> 'dp-consumption-aggregation-processor-staging2' is still active. Make > >> sure > >>> to stop all running application instances before running the reset > tool. > >>> java.lang.IllegalStateException: Consumer group > >>> 'dp-consumption-aggregation-processor-staging2' is still active. Make > >> sure > >>> to stop all running application instances before running the reset > tool. > >>> > >>> But as I said the application is not running. So I ran > >>> > >>> ./kafka-consumer-groups.sh --bootstrap-server > >>> smf1-abi-09-sr1.prod.twitter.com:9092 --group > >>> dp-consumption-aggregation-processor-staging2 --describe > >>> > >>> and the first line is > >>> > >>> Warning: Consumer group 'dp-consumption-aggregation- > processor-staging2' > >> is > >>> rebalancing. > >>> > >>> How do I get it out of this state and how did I get it into this state? > >>> > >>> Thanks! > >>> > >> > >> > > > > > > -- -- Russell Teabeault | Senior Software Engineer | Twitter | @rusticules
