*if the consuming of all produced data is not important On Tue, Oct 29, 2019 at 12:42 PM Daniyar Kulakhmetov < [email protected]> wrote:
> As a solution, you may have multiple Kafka clusters. For example, 1 main > Kafka cluster + 7 secondary (mirrored) Kafka clusters. > The logic would be as follows: > 1) Messages are produced to the main Kafka cluster > 2) Secondary Kafka clusters consume data from the main Kafka cluster > 3) 7K consumers are split between secondary Kafka clusters and consume > data from them (~1000 consumers per 1 secondary Kafka cluster). > > In this case, the consuming load would be distributed. > > As an alternative, if the consuming of all produced data, you can you > Kafka + Rabbit MQ fanout exchange. > > > > > On Wed, Oct 23, 2019 at 4:41 AM Hrishikesh Mishra <[email protected]> > wrote: > >> I have following Kafka cluster: >> >> >> - Broker#: 13 (1 Broker : 14 cores & 36GB memory ) >> - Kafka cluster version: 2.0.0 >> - Kafka Java client version: 2.0.0 >> - Number topics: ~15 (3 replica, 2 min replica, 8 partitions) >> - Number of consumers: 7K (all independent and manually assigned all >> partitions of a topic to a consumers. One consumer is consuming all >> partitions from a topic only) >> >> >> >> - CPU utilisation (per broker): ~15% >> - Memory Utilisation: ~ 20% >> - Bytes Out: ~30MBps >> - Message produce rate: ~20K >> >> >> Message is very small and produce rate very low. That's why there is no >> load on brokers. I wanted to understand of having large number of >> consumers >> (~300 to ~500) consuming independently (manual partition assigned) for a >> topic then what problem can come in future like: >> >> - will there lag build up when spike come and Kafka starts not behaving >> properly? >> - Presently, I am not able to open kafka manager to check lag because >> it >> getting timeout due to huge number of consumer per topic. >> - Some errors are also logged but as a INFO. >> >> >> INFO [FetchSessionHandler:440] [Consumer clientId=XXXX#2321d06ae56a#test, >> groupId=XXXX#2321d06ae56a#test] Error sending fetch request >> (sessionId=1534027518, epoch=1) to node 4: >> org.apache.kafka.common.errors.DisconnectException.17:09:09,161 INFO >> [FetchSessionHandler:440] [Consumer clientId=XXXX#2321d06ae56a#test, >> groupId=XXXX#2321d06ae56a#test] Error sending fetch request >> (sessionId=1999034475, epoch=INITIAL) to node 5: >> org.apache.kafka.common.errors.DisconnectException.17:09:09,161 INFO >> [FetchSessionHandler:440] [Consumer clientId=XXXX#2321d06ae56a#test, >> groupId=XXXX#2321d06ae56a#test] Error sending fetch request >> (sessionId=1788862064, epoch=1) to node 9: >> org.apache.kafka.common.errors.DisconnectException. >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Tue, Oct 22, 2019 at 2:50 PM M. Manna <[email protected]> wrote: >> >> > Everything has impact. You cannot keep churning loads of messages under >> the >> > same operating condition, and expect nothing to change. >> > >> > You have know find out (via load testing) an optimum operating condition >> > (e.g. partition, batch.size etc.) for you producer/consumer to work >> > correctly. Remember that more topics/partition you have, the more the >> > complexity. Based on how many topics/partitions/consumers/producers >> you're >> > creating - the tuning of brokers may need to change accordingly. >> > >> > Thanks, >> > >> > On Tue, 22 Oct 2019 at 10:09, Hrishikesh Mishra <[email protected]> >> > wrote: >> > >> > > I wanted to understand whether broker will be unstable with large >> number >> > of >> > > consumers or will consume face some issue like lag will increase? >> > > >> > > >> > > On Mon, Oct 21, 2019 at 6:55 PM Shyam P <[email protected]> >> wrote: >> > > >> > > > What are you trying to do here ? whats your objective ? >> > > > >> > > > On Sat, Oct 19, 2019 at 8:45 PM Hrishikesh Mishra < >> [email protected] >> > > >> > > > wrote: >> > > > >> > > > > Can anyone please help me with this? >> > > > > >> > > > > On Fri, 18 Oct 2019 at 2:58 PM, Hrishikesh Mishra < >> > [email protected] >> > > > >> > > > > wrote: >> > > > > >> > > > > > Hi all, >> > > > > > >> > > > > > I wanted to understand, having large numbers of consumers on >> > > > > > producer latency and brokers. I have around 7K independent >> > consumers. >> > > > > Each >> > > > > > consumer is consuming all partitions of a topic. I have manually >> > > > assigned >> > > > > > partitions of a topic to a consumer, not using consumer groups. >> > Each >> > > > > > consumer is consuming msg only one topic. Message size is small >> > less >> > > > > than a >> > > > > > KB. Produce throughput is also low. But when produce spike come >> > then >> > > > > > produce latency (here ack=1). Brokers resource is also very >> low. I >> > > want >> > > > > to >> > > > > > understand the impact on having large number of consumers on >> Kafka. >> > > > > > >> > > > > > >> > > > > > *Cluster Details: * >> > > > > > >> > > > > > Broker#: 13 (1 Broker : 14 cores & 36GB memory ) >> > > > > > >> > > > > > Kafka cluster version: 2.0.0 >> > > > > > Kafka Java client version: 2.0.0 >> > > > > > Number topics: ~15. >> > > > > > Number of consumers: 7K (all independent and manually assigned >> all >> > > > > > partitions of a topic to a consumers. One consumer is consuming >> all >> > > > > > partitions from a topic only) >> > > > > > >> > > > > > >> > > > > > >> > > > > > Regards, >> > > > > > Hrishikesh >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
