Hi Paras, Could you share more information about your setup:
* What's the alerting rule that isn't working as intended? * For how long were the hosts down without getting alerted on? * What did the underlying metrics (e.g. "up" for the exporter's own scrape health and "probe_success" for the backend probe health) collected by the Blackbox Exporter look like at the time when the alert should have been firing, but didn't? One possibility is that your Blackbox exporter itself couldn't be scraped anymore, in which case its "up" metric would be 0 and the "probe_success" metric would be absent (and thus any alerts based on that metric would never fire). Regards, Julius On Thu, Sep 15, 2022 at 6:33 PM Paras pradhan <[email protected]> wrote: > Hello, > > We use prometheus , alertmanager and blackbox-exporter to check hosts if > they respond to icmp. Host counts are 1K+. We noticed sometimes and > randomly the alerts are not generated (prometheus dashboard --> alerts) > when the hosts/targets are actually down. Restarting prometheus, > alertmanager and blackbox-exports fixes the issue. Don't see anything that > standouts in the logs. How do I troubleshoot and is there anything like > cache data in prometheus that needs to be cleared? > > Thanks > Paras. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/6bfb92dc-2a18-44d9-8fda-d6f84efba0e7n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/6bfb92dc-2a18-44d9-8fda-d6f84efba0e7n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5yqfCirzSvZLmJy0rQfv8qTERWWwYzjZrMJ5YubveiN0A%40mail.gmail.com.
