Everything has impact. You cannot keep churning loads of messages under the same operating condition, and expect nothing to change.
You have know find out (via load testing) an optimum operating condition (e.g. partition, batch.size etc.) for you producer/consumer to work correctly. Remember that more topics/partition you have, the more the complexity. Based on how many topics/partitions/consumers/producers you're creating - the tuning of brokers may need to change accordingly. Thanks, On Tue, 22 Oct 2019 at 10:09, Hrishikesh Mishra <[email protected]> wrote: > I wanted to understand whether broker will be unstable with large number of > consumers or will consume face some issue like lag will increase? > > > On Mon, Oct 21, 2019 at 6:55 PM Shyam P <[email protected]> wrote: > > > What are you trying to do here ? whats your objective ? > > > > On Sat, Oct 19, 2019 at 8:45 PM Hrishikesh Mishra <[email protected]> > > wrote: > > > > > Can anyone please help me with this? > > > > > > On Fri, 18 Oct 2019 at 2:58 PM, Hrishikesh Mishra <[email protected] > > > > > wrote: > > > > > > > Hi all, > > > > > > > > I wanted to understand, having large numbers of consumers on > > > > producer latency and brokers. I have around 7K independent consumers. > > > Each > > > > consumer is consuming all partitions of a topic. I have manually > > assigned > > > > partitions of a topic to a consumer, not using consumer groups. Each > > > > consumer is consuming msg only one topic. Message size is small less > > > than a > > > > KB. Produce throughput is also low. But when produce spike come then > > > > produce latency (here ack=1). Brokers resource is also very low. I > want > > > to > > > > understand the impact on having large number of consumers on Kafka. > > > > > > > > > > > > *Cluster Details: * > > > > > > > > Broker#: 13 (1 Broker : 14 cores & 36GB memory ) > > > > > > > > Kafka cluster version: 2.0.0 > > > > Kafka Java client version: 2.0.0 > > > > Number topics: ~15. > > > > Number of consumers: 7K (all independent and manually assigned all > > > > partitions of a topic to a consumers. One consumer is consuming all > > > > partitions from a topic only) > > > > > > > > > > > > > > > > Regards, > > > > Hrishikesh > > > > > > > > > > > > > >
