It's maybe worth mentioning that alert thresholds can come from their own timeseries, as described here: https://www.robustperception.io/using-time-series-as-alert-thresholds
That can help automate these updates. You don't need to modify any rules; instead you expose the 'current' value of the threshold, let it be scraped as normal, and modify the threshold as required. As a bonus, you get a full history of which thresholds were used and when. But I don't know of a complete out-of-box solution which provides the full feedback loop. On Wednesday, 29 September 2021 at 06:33:55 UTC+1 [email protected] wrote: > Hello, > No I am not aware of such tool, but it shouldnt be hard to write a simple > exporter (maybe using python prometheus_client lib) to replay historical > data and expose it to a Prometheus/Alertmanager setup. > Or, given the alerts are also stored in the TSDB, you can build sth that > navigate the data on time basis and detect when the state of the alert > changed to "pending" or "firing" and check the thresholds > > On Tuesday, September 28, 2021 at 10:20:31 AM UTC+2 [email protected] > wrote: > >> I'm thinking about ways we can reduce noisy alerts. One of the problems >> is it's tricky to tweak alert thresholds without any data on the precision >> and recall of the alert. It's a non-trivial problem to get this data >> because a human is typically required to classify an alert as a true >> positive or a false negative. This makes it hard to fully automate >> gathering this data. I am considering whether there is a way of obtaining >> this data using a hybrid approach: a human is able to classify an alert as >> a true positive of false positive - for example via a button in the alert >> body (e.g. in Slack or PagerDuty) and this gets sent to an analytics >> database which we can later prioritise which alert thresholds that need >> tweaking. >> >> My question is, is there any precedent for this kind of system in >> the Prometheus/Alertmanager ecosystem? i.e. open source software that does >> this out of the box, or experience report blog posts? >> >> Many thanks, >> Will >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c2bebe19-55c6-4510-a0a9-70ee932f3ae6n%40googlegroups.com.
