Hello Dev community, Got no response from user community on below query. Can you respond back on this please?
---------- Forwarded message --------- From: Pushkar Deole <pdeole2...@gmail.com> Date: Fri, Jan 17, 2020 at 1:46 PM Subject: Is there a way to auto scale topic partitions in kafka? To: <us...@kafka.apache.org> Hello, I am working on developing a microservice based system which uses kafka as a messaging infrastructure. The microservices application are mainly kafka consumers and kafka streams applications and are deployed as docker containers on kubernetes. The system should be designed to be auto scalable for which we are using Horizontal Pod Autoscaler feature of kubernetes which allows to instantiate more number of pods if a certain metric (e.g. cpu utilization) touches the threshold or reduce the pods in case the metric is way below the threshold. However, the problem is number of partitions in kafka are fixed so even if load on the system increases and the number of consumer pods are autoscaled, it could not be scaled beyond the number of partitions. So, after a point where number of pods is equal to number of partitions, the system can't be scaled beyond that. Is there a way to autoscale number of partitions also in kafka so the system can be auto scaled in cloud?
