Kafka alone wouldn't scale down your consumers, but you can get metrics from Kafka and use them to change the number of consumers.
Scaling up/down is normally done at other app levels, using tools whose purpose is to watch metrics and scaling up/down (say HPA kubernetes / openshift if you are working with docker containers, or Cloud watch ASG in AWS). Lightbend has an open source consumer lag exporter that could help you exposing Kafka consumer metrics (prometheus style), for instance. But there are several other ways to get them. On Sun, Nov 15, 2020, 7:09 AM Mazen Ezzeddine <mazen.ezzedd...@gmail.com> wrote: > Thanks, > My question is mostly about dynamic resource optimization, > > say I configured my application with 30 partitions and then I managed for > 30 consumers (within a consumer group) to read/process the produced > messages, but say at for instance at some unpeak load, I realized that a > single consumer (within the same consumer group) could do consume all > messages produced (production rate is low)? are you aware of any Kafka > solution that would automatically join/leave consumers from within a > consumer group dynamically e.g., based on how well the consumer group is > doing to keep up with the production rate? > > On 2020/11/14 22:36:34, Vinicius Scheidegger < > vinicius.scheideg...@gmail.com> wrote: > > This depends on the design of your application and how you are using > Kafka. > > > > For instance, if you are using Kafka as a message queuing app, using > > consumers within a consumer group to load balance, then you should create > > the topics with as many partitions as the max number of consumers within > > the same consumer group reading from it. > > > > For instance, in a regular load you believe you will have 8 consumers > > reading from a topic, but at peak you believe you could have around 30 > > consumers. You should then have at least 30 partitions - each consumer > in a > > consumer group is assigned to a partition. Having more consumers in a > > single consumer group than partitions are not effective as they won't be > > receiving any messages. > > > > But as I mentioned earlier different designs may lead to different > > decisions. Try to understand first how partitions work. How they are > > assigned in your context (there are different partitioning schemes) and > > then you may have a better sense of it. > > > > I hope it helps > > > > Vinicius Scheidegger > > > > > > > > On Sat, Nov 14, 2020, 8:39 PM Mazen Ezzeddine < > > mazen.ezzedd...@etu.univ-cotedazur.fr> wrote: > > > > > Given a business application that resorts into a message queue solution > > > like Kafka, what is the best number of partitions to select for a given > > > topic? what influences such a decision? > > > > > > > > > On the other hand, say we want to achieve a maximal throughput of > message > > > consumption but at minimal resource consumption? what is the best > number of > > > topic consumers to configure statically? > > > > > > > > > > > > If dynamic scale up/down of topic consumers is enabled what would > better : > > > (1) start with one consumer and scale up consumers until a desired > metric > > > is achieved, or (2) start with consumers equal to the number of > partitions > > > and then scale down until the desired metric is achieved? > > > > > > > > > Are you aware of any cloud provider that offers a message broker > service > > > (namely, Kafka) that support automatic scale of consumers? > > > > > > > > > > > > Thank you. > > > > > > > > >
