This depends on the design of your application and how you are using Kafka.
For instance, if you are using Kafka as a message queuing app, using consumers within a consumer group to load balance, then you should create the topics with as many partitions as the max number of consumers within the same consumer group reading from it. For instance, in a regular load you believe you will have 8 consumers reading from a topic, but at peak you believe you could have around 30 consumers. You should then have at least 30 partitions - each consumer in a consumer group is assigned to a partition. Having more consumers in a single consumer group than partitions are not effective as they won't be receiving any messages. But as I mentioned earlier different designs may lead to different decisions. Try to understand first how partitions work. How they are assigned in your context (there are different partitioning schemes) and then you may have a better sense of it. I hope it helps Vinicius Scheidegger On Sat, Nov 14, 2020, 8:39 PM Mazen Ezzeddine < mazen.ezzedd...@etu.univ-cotedazur.fr> wrote: > Given a business application that resorts into a message queue solution > like Kafka, what is the best number of partitions to select for a given > topic? what influences such a decision? > > > On the other hand, say we want to achieve a maximal throughput of message > consumption but at minimal resource consumption? what is the best number of > topic consumers to configure statically? > > > > If dynamic scale up/down of topic consumers is enabled what would better : > (1) start with one consumer and scale up consumers until a desired metric > is achieved, or (2) start with consumers equal to the number of partitions > and then scale down until the desired metric is achieved? > > > Are you aware of any cloud provider that offers a message broker service > (namely, Kafka) that support automatic scale of consumers? > > > > Thank you. > >
