As a solution, you may have multiple Kafka clusters. For example, 1 main Kafka cluster + 7 secondary (mirrored) Kafka clusters. The logic would be as follows: 1) Messages are produced to the main Kafka cluster 2) Secondary Kafka clusters consume data from the main Kafka cluster 3) 7K consumers are split between secondary Kafka clusters and consume data from them (~1000 consumers per 1 secondary Kafka cluster).
In this case, the consuming load would be distributed. As an alternative, if the consuming of all produced data, you can you Kafka + Rabbit MQ fanout exchange. On Wed, Oct 23, 2019 at 4:41 AM Hrishikesh Mishra <[email protected]> wrote: > I have following Kafka cluster: > > > - Broker#: 13 (1 Broker : 14 cores & 36GB memory ) > - Kafka cluster version: 2.0.0 > - Kafka Java client version: 2.0.0 > - Number topics: ~15 (3 replica, 2 min replica, 8 partitions) > - Number of consumers: 7K (all independent and manually assigned all > partitions of a topic to a consumers. One consumer is consuming all > partitions from a topic only) > > > > - CPU utilisation (per broker): ~15% > - Memory Utilisation: ~ 20% > - Bytes Out: ~30MBps > - Message produce rate: ~20K > > > Message is very small and produce rate very low. That's why there is no > load on brokers. I wanted to understand of having large number of consumers > (~300 to ~500) consuming independently (manual partition assigned) for a > topic then what problem can come in future like: > > - will there lag build up when spike come and Kafka starts not behaving > properly? > - Presently, I am not able to open kafka manager to check lag because it > getting timeout due to huge number of consumer per topic. > - Some errors are also logged but as a INFO. > > > INFO [FetchSessionHandler:440] [Consumer clientId=XXXX#2321d06ae56a#test, > groupId=XXXX#2321d06ae56a#test] Error sending fetch request > (sessionId=1534027518, epoch=1) to node 4: > org.apache.kafka.common.errors.DisconnectException.17:09:09,161 INFO > [FetchSessionHandler:440] [Consumer clientId=XXXX#2321d06ae56a#test, > groupId=XXXX#2321d06ae56a#test] Error sending fetch request > (sessionId=1999034475, epoch=INITIAL) to node 5: > org.apache.kafka.common.errors.DisconnectException.17:09:09,161 INFO > [FetchSessionHandler:440] [Consumer clientId=XXXX#2321d06ae56a#test, > groupId=XXXX#2321d06ae56a#test] Error sending fetch request > (sessionId=1788862064, epoch=1) to node 9: > org.apache.kafka.common.errors.DisconnectException. > > > > > > > > > > > > > > On Tue, Oct 22, 2019 at 2:50 PM M. Manna <[email protected]> wrote: > > > Everything has impact. You cannot keep churning loads of messages under > the > > same operating condition, and expect nothing to change. > > > > You have know find out (via load testing) an optimum operating condition > > (e.g. partition, batch.size etc.) for you producer/consumer to work > > correctly. Remember that more topics/partition you have, the more the > > complexity. Based on how many topics/partitions/consumers/producers > you're > > creating - the tuning of brokers may need to change accordingly. > > > > Thanks, > > > > On Tue, 22 Oct 2019 at 10:09, Hrishikesh Mishra <[email protected]> > > wrote: > > > > > I wanted to understand whether broker will be unstable with large > number > > of > > > consumers or will consume face some issue like lag will increase? > > > > > > > > > On Mon, Oct 21, 2019 at 6:55 PM Shyam P <[email protected]> > wrote: > > > > > > > What are you trying to do here ? whats your objective ? > > > > > > > > On Sat, Oct 19, 2019 at 8:45 PM Hrishikesh Mishra < > [email protected] > > > > > > > wrote: > > > > > > > > > Can anyone please help me with this? > > > > > > > > > > On Fri, 18 Oct 2019 at 2:58 PM, Hrishikesh Mishra < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I wanted to understand, having large numbers of consumers on > > > > > > producer latency and brokers. I have around 7K independent > > consumers. > > > > > Each > > > > > > consumer is consuming all partitions of a topic. I have manually > > > > assigned > > > > > > partitions of a topic to a consumer, not using consumer groups. > > Each > > > > > > consumer is consuming msg only one topic. Message size is small > > less > > > > > than a > > > > > > KB. Produce throughput is also low. But when produce spike come > > then > > > > > > produce latency (here ack=1). Brokers resource is also very low. > I > > > want > > > > > to > > > > > > understand the impact on having large number of consumers on > Kafka. > > > > > > > > > > > > > > > > > > *Cluster Details: * > > > > > > > > > > > > Broker#: 13 (1 Broker : 14 cores & 36GB memory ) > > > > > > > > > > > > Kafka cluster version: 2.0.0 > > > > > > Kafka Java client version: 2.0.0 > > > > > > Number topics: ~15. > > > > > > Number of consumers: 7K (all independent and manually assigned > all > > > > > > partitions of a topic to a consumers. One consumer is consuming > all > > > > > > partitions from a topic only) > > > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > Hrishikesh > > > > > > > > > > > > > > > > > > > > > > > > > > >
