Best Practice Scaling Consumers

Hi all, I’m new to Kafka and have a very basic question: We build a cloud-scale platform and evaluate if we can use Kafka for pub-sub messaging between our services. Most of our services scale dynamically based on load (number of requests, CPU load etc.). In our current architecture, services

Re: Best Practice Scaling Consumers

1. Yes, you may have to overprovision the number of partitions to handle the load peaks. Refer this document to choose the no. of partitions. 2. KIP-429

Re: Best Practice Scaling Consumers

Hi Morritz - I don’t believe the number of Kafka consumers is restricted to the number of partitions. When you create a topic - and indicate both the number of partitions and a key - it causes your key value pairs to be allocated to a specific partition on the basis of a hash function on the