Thanks Liam, I am trying hpa but using cpu utilization, but since everything is tied to partition number etc i wonder what the benefits of running on hpa really are.
thanks! On Mon, Feb 28, 2022 at 12:59 PM Liam Clarke-Hutchinson <lclar...@redhat.com> wrote: > I've used HPAs scaling on lag before by feeding lag metrics from Prometheus > into the K8s metrics server as custom metrics. > > That said, you need to carefully control scaling frequency to avoid > excessive consumer group rebalances. The cooperative sticky assignor can > minimise pauses, but not remove them entirely. > > There's a lot of knobs you can use to tune HPAs these days: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__kubernetes.io_docs_tasks_run-2Dapplication_horizontal-2Dpod-2Dautoscale_-23configurable-2Dscaling-2Dbehavior&d=DwIBaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=dQzp4x9JZe-7YZcgrSl3YrB3X7PYTM_bS4caOQ59hLLonNXE0x3TveYTXVAFcxco&s=_NU3o8FG8CwNpe8wl3mVxXkNeEx_9aCD2_md1riEZa0&e= > > Good luck :) > > > > On Tue, 1 Mar 2022 at 08:49, David Ballano Fernandez < > dfernan...@demonware.net> wrote: > > > Hello Guys, > > > > I was wondering how you guys do autoscaling of you consumers in > kubernetes > > if you do any. > > > > We have a mirrormaker-like app that mirrors data from cluster to cluster > at > > the same time does some topic routing. I would like to add hpa to the > app > > in order to scale up/down depending on avg cpu. but as you know a > consumer > > app has lots of variables being partitions of topics being consumed a > > pretty important one. > > > > Since kubernetes checks cpu avg, there are chances that pods/consumers > > won't be scaled up to the number of partitions possibly creating some > hot > > spots. > > > > Anyways i would like to know how you deal if you do at all with this. > > > > thanks! > > >
