Hello You can look at keda.sh which allows to auto scale workloads on kubernetes according to many custom metrics including Kafka lag for instance.
Fares Le mer. 2 mars 2022 à 01:07, David Ballano Fernandez < dfernan...@demonware.net> a écrit : > Thanks Liam, > > I am trying hpa but using cpu utilization, but since everything is tied to > partition number etc i wonder what the benefits of running on hpa really > are. > > thanks! > > On Mon, Feb 28, 2022 at 12:59 PM Liam Clarke-Hutchinson < > lclar...@redhat.com> > wrote: > > > I've used HPAs scaling on lag before by feeding lag metrics from > Prometheus > > into the K8s metrics server as custom metrics. > > > > That said, you need to carefully control scaling frequency to avoid > > excessive consumer group rebalances. The cooperative sticky assignor can > > minimise pauses, but not remove them entirely. > > > > There's a lot of knobs you can use to tune HPAs these days: > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__kubernetes.io_docs_tasks_run-2Dapplication_horizontal-2Dpod-2Dautoscale_-23configurable-2Dscaling-2Dbehavior&d=DwIBaQ&c=qE8EibqjfXM-zBfebVhd4gtjNZbrDcrKYXvb1gt38s4&r=p-f3AJg4e4Uk20g_16kSyBtabT4JOB-1GIb23_CxD58&m=dQzp4x9JZe-7YZcgrSl3YrB3X7PYTM_bS4caOQ59hLLonNXE0x3TveYTXVAFcxco&s=_NU3o8FG8CwNpe8wl3mVxXkNeEx_9aCD2_md1riEZa0&e= > > > > Good luck :) > > > > > > > > On Tue, 1 Mar 2022 at 08:49, David Ballano Fernandez < > > dfernan...@demonware.net> wrote: > > > > > Hello Guys, > > > > > > I was wondering how you guys do autoscaling of you consumers in > > kubernetes > > > if you do any. > > > > > > We have a mirrormaker-like app that mirrors data from cluster to > cluster > > at > > > the same time does some topic routing. I would like to add hpa to the > > app > > > in order to scale up/down depending on avg cpu. but as you know a > > consumer > > > app has lots of variables being partitions of topics being consumed a > > > pretty important one. > > > > > > Since kubernetes checks cpu avg, there are chances that pods/consumers > > > won't be scaled up to the number of partitions possibly creating some > > hot > > > spots. > > > > > > Anyways i would like to know how you deal if you do at all with this. > > > > > > thanks! > > > > > >
