I've used HPAs scaling on lag before by feeding lag metrics from Prometheus into the K8s metrics server as custom metrics.
That said, you need to carefully control scaling frequency to avoid excessive consumer group rebalances. The cooperative sticky assignor can minimise pauses, but not remove them entirely. There's a lot of knobs you can use to tune HPAs these days: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#configurable-scaling-behavior Good luck :) On Tue, 1 Mar 2022 at 08:49, David Ballano Fernandez < dfernan...@demonware.net> wrote: > Hello Guys, > > I was wondering how you guys do autoscaling of you consumers in kubernetes > if you do any. > > We have a mirrormaker-like app that mirrors data from cluster to cluster at > the same time does some topic routing. I would like to add hpa to the app > in order to scale up/down depending on avg cpu. but as you know a consumer > app has lots of variables being partitions of topics being consumed a > pretty important one. > > Since kubernetes checks cpu avg, there are chances that pods/consumers > won't be scaled up to the number of partitions possibly creating some hot > spots. > > Anyways i would like to know how you deal if you do at all with this. > > thanks! >
