When doing autoscaling (not specifically with Prometheus but everything) you need to ensure that you don't have too many changes happening at once, otherwise you might start rejecting requests (if all instances are restarting at the same time).
This would generally be done via things like pod distuption budgets. For a pair of Prometheus servers I'd not want more than one change at once. For other systems I might go as far as N-1 changes at once. On Monday, 7 June 2021 at 11:05:20 UTC+1 Stuart Clark wrote: > On 2021-06-07 10:36, nina guo wrote: > > Autoscaling based on the prometheus metrics load. > > For example, if during the recent minutes, the metrics is more than > > 50m, a new prometheus pod will be started. > > > > So you are wanting to autoscale Prometheus itself? That's not normally > needed, as the load should be pretty even across instances (assuming > some sort of load balancing or dedup process in front that handles > queries) as each instance will be scraping the same targets. If the > number of targets/time series increases you might need to increase the > memory usage (so vertical pod autoscaling of the two instances rather > than horizontal pod autoscaling). Similarly for queries it would be more > memory you might need mostly rather than additional instances - if you > are expecting massive query load maybe something like Thanos might be a > good option? > > -- > Stuart Clark > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/48618411-f84f-4cca-b950-027fdac32b6cn%40googlegroups.com.
