Hello, No I am not aware of such tool, but it shouldnt be hard to write a simple exporter (maybe using python prometheus_client lib) to replay historical data and expose it to a Prometheus/Alertmanager setup. Or, given the alerts are also stored in the TSDB, you can build sth that navigate the data on time basis and detect when the state of the alert changed to "pending" or "firing" and check the thresholds
On Tuesday, September 28, 2021 at 10:20:31 AM UTC+2 [email protected] wrote: > I'm thinking about ways we can reduce noisy alerts. One of the problems is > it's tricky to tweak alert thresholds without any data on the precision and > recall of the alert. It's a non-trivial problem to get this data because a > human is typically required to classify an alert as a true positive or a > false negative. This makes it hard to fully automate gathering this data. I > am considering whether there is a way of obtaining this data using a hybrid > approach: a human is able to classify an alert as a true positive of false > positive - for example via a button in the alert body (e.g. in Slack or > PagerDuty) and this gets sent to an analytics database which we can later > prioritise which alert thresholds that need tweaking. > > My question is, is there any precedent for this kind of system in > the Prometheus/Alertmanager ecosystem? i.e. open source software that does > this out of the box, or experience report blog posts? > > Many thanks, > Will > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/95d8ce9b-4354-4a5e-b1c2-9abb051ffc66n%40googlegroups.com.
