On 22 Jul 02:14, Lili Cosic wrote: > Only now seen in the docs that I am supposed to start any discussions here > first before opening an issue, sorry about that! :) > > Currently there is no way of a target to have higher scrape priority over > another, but if you have a setup and even if you set target limits and > sample limits you can still overestimate your setup, you still want to have > a higher priority targets that are preferred over the entire Prometheus to > fail. It would need to be based on the inability to ingest into tsdb on the > current rate we are scrapping, if that is hit the priority class would take > affect and only the highest priority targets would be scrapped in favour of > lower priority. Another option which might be simpler would be to have a > global limit on how much prometheus can handle based on perf testing. > > This would be treated as a last resort, and there would definitely be a > need for a high severity alert to inform the admin that something went > terribly wrong, but because we would still be able to ingest Prometheus > metrics for example if they are higher priority class alerting would be > possible.
Hi, I think that limiting the number of targets you scrape is already a last resort. I don't think we would need a second line of defense. You can achieve this priority by setting 2 jobs, one which is limited and one which is not, and use relabeling to decinde which target is going in which job. > > We could model this on something like PriorityClass > <https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass> > from > Kubernetes, but I am open to other suggestions. That could be used in relabeling as I said. > > I am open to other suggestions, or maybe there is something like this but I > missed it. The main purpose is to ensure there are protection mechanisms in > place, so any ideas and suggestions welcome! > regards, > Thanks and kind regards, > Lili > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/30df615e-5420-4bdf-9cb7-2790ef19d520o%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200722091759.GA140540%40oxygen.
