Hi about question 1, it's dosen't matter that how many consumers in same consumer group.
So you means the broker which is coordinator did not crashed at all before? May i know if only exact one broker(coordinator) is unavailable or many are? if only exact one, you can try to transfer leader of _consumer_offset which on that broker to another broker to see if it's no problem any more? i found the following issue seems similar with yours, FYR: https://stackoverflow.com/questions/51952398/kafka-connect-distributed-mode-the-group-coordinator-is-not-available Best, Lisheng Hrishikesh Mishra <sd.hri...@gmail.com> 于2019年8月29日周四 下午12:19写道： > Hi, > > We are facing following issues with Kafka cluster. > > - Kafka Version: 2.0.0 > - We following cluster configuration: > - Number of Broker: 14 > - Per Broker: 37GB Memory and 14 Cores. > - Topics: 40 - 50 > - Partitions per topic: 32 > - Replicas: 3 > - Min In Sync Replica: 2 > - __consumer_topic partition: 50 > - offsets.topic.replication.factor=3 > - default.replication.factor=3 > - Consumers#: ~4000 (will grow to ~7K) > - Consumer Groups#: ~4000 (will grow to ~7K) > > > Imp: Here one consumer is consuming from one topic and one consumer group > has only one consumer due to some architectural constraints. > > Two major problems we are facing with consumer group: > > - First time when we are starting consumer with new group name it > working very well. But subsequent restart (with previous / older group > name) is causing problems from some consumers. We are getting following > errors: > > INFO [2019-08-28 19:05:34,481] [main] [AbstractCoordinator]: [Consumer > clientId=djXXX#XXX-XXX-XXX-XX-5-1478729-XX-XXXXX-XX-ingestion-v2, > groupId=djXXX#XXX-XXX-XXX-XX-5-1478729-XX-XXXXX-XX-ingestion-v2] > Discovered > group coordinator 10.XX.XXX.112:9092 (id: 2147483631 rack: null) > INFO [2019-08-28 19:05:34,481] [main] [AbstractCoordinator]: [Consumer > clientId=djXXX#XXX-XXX-XXX-XX-5-1478729-XX-XXXXX-XX-ingestion-v2, > groupId=djXXX#XXX-XXX-XXX-XX-5-1478729-XX-XXXXX-XX-ingestion-v2] Group > coordinator 10.XX.XXX.112:9092 (id: 2147483631 rack: null) is > unavailable > or invalid, will attempt rediscovery > INFO [2019-08-28 19:05:34,582] [main] [AbstractCoordinator]: [Consumer > clientId=djXXX#XXX-XXX-XXX-XX-5-1478729-XX-XXXXX-XX-ingestion-v2, > groupId=djXXX#XXX-XXX-XXX-XX-5-1478729-XX-XXXXX-XX-ingestion-v2] > Discovered > group coordinator 10.32.197.112:9092 (id: 2147483631 rack: null) > > These messages are keep coming and consumer not able to start / poll. > But if we change the group name then it works first time without any > issue > (and fails in subsequent restart). So it also means that there is no > with > issue broker. Will it because of having single consumer in consumer > group, > if yes then what will be the work around here? > > - The second error, we are getting when consumer is up and running. Then > after couple hours, it starts failing and throwing following error: > Consumer clientId=banneXXXX#XX-XXX-XXX-XXX-X-1388688-XXX-XXXXX, > groupId=bannerXXX#XX-XXX-XXX-XXX-X-1388688-XXX-XXXXX] Offset commit > failed > on partition banneXXXX-7 at offset 13711176: This is not the correct > coordinator > [Consumer > > clientId=banXXerGrXXMXX#XX-XX-XXXXX-XXX-5-1478733-XXX-XXXXX-ingestion-v2, > groupId=banXXerGrXXMXX#XX-XX-XXXXX-XXX-5-1478733-XXX-XXXXX-ingestion-v2] > Offset commit failed on partition banXXerGrXXMXX-8 at offset 14741: > This is > not the correct coordinator. > > > I wanted to know following things: > > - What is the max limit of consumer groups in a Kafka cluster, I didn't > find any limitation on internet, all places it mentioned that limited > by OS. > - Is there a problem of a consumer group has only one consumer. > - Is there some problem with my Kafka configuration, > > > > > Regards > Hrishikesh >
