On Wednesday, 22 July 2020 11:23:00 UTC+2, Brian Brazil wrote: > > On Wed, 22 Jul 2020 at 10:18, Julien Pivotto <roidel...@prometheus.io > <javascript:>> wrote: > >> On 22 Jul 02:14, Lili Cosic wrote: >> > Only now seen in the docs that I am supposed to start any discussions >> here >> > first before opening an issue, sorry about that! :) >> > >> > Currently there is no way of a target to have higher scrape priority >> over >> > another, but if you have a setup and even if you set target limits and >> > sample limits you can still overestimate your setup, you still want to >> have >> > a higher priority targets that are preferred over the entire Prometheus >> to >> > fail. It would need to be based on the inability to ingest into tsdb on >> the >> > current rate we are scrapping, if that is hit the priority class would >> take >> > affect and only the highest priority targets would be scrapped in >> favour of >> > lower priority. Another option which might be simpler would be to have >> a >> > global limit on how much prometheus can handle based on perf testing. >> > >> > This would be treated as a last resort, and there would definitely be a >> > need for a high severity alert to inform the admin that something went >> > terribly wrong, but because we would still be able to ingest Prometheus >> > metrics for example if they are higher priority class alerting would be >> > possible. >> >> Hi, >> >> I think that limiting the number of targets you scrape is already a last >> resort. I don't think we would need a second line of defense. >> > > I agree with Julien here. If you've gotten to this point you're already > seriously overloaded, and prioritising individual targets is just > rearranging the deckchairs at that point. > > >> >> You can achieve this priority by setting 2 jobs, one which is limited >> and one which is not, and use relabeling to decinde which target is >> going in which job. >> > > Or more generally, one Prometheus for the important targets and another > for the less important and riskier targets. >
I get your point completely Brian, and agree to some degree but people are still going to be setting up a multi tenant prometheus which then causes the above problems I mentioned. Even within the riskier targets there will be some more important than others for users. I think we should still strive to making a single shared Prometheus as safe as possible, if this is not the priority class I suggested, open to other ideas! > > Brian > > >> >> > >> > We could model this on something like PriorityClass >> > < >> https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass> >> >> from >> > Kubernetes, but I am open to other suggestions. >> >> That could be used in relabeling as I said. >> >> > >> > I am open to other suggestions, or maybe there is something like this >> but I >> > missed it. The main purpose is to ensure there are protection >> mechanisms in >> > place, so any ideas and suggestions welcome! >> > >> >> regards, >> >> > Thanks and kind regards, >> > Lili >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Developers" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to prometheus-developers+unsubscr...@googlegroups.com >> <javascript:>. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/30df615e-5420-4bdf-9cb7-2790ef19d520o%40googlegroups.com >> . >> >> >> -- >> Julien Pivotto >> @roidelapluie >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to prometheus-developers+unsubscr...@googlegroups.com <javascript:> >> . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-developers/20200722091759.GA140540%40oxygen >> . >> > > > -- > Brian Brazil > www.robustperception.io > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b0b9e5f7-239a-4cc7-9108-9e6e015a30d6o%40googlegroups.com.
