Hello, We use prometheus , alertmanager and blackbox-exporter to check hosts if they respond to icmp. Host counts are 1K+. We noticed sometimes and randomly the alerts are not generated (prometheus dashboard --> alerts) when the hosts/targets are actually down. Restarting prometheus, alertmanager and blackbox-exports fixes the issue. Don't see anything that standouts in the logs. How do I troubleshoot and is there anything like cache data in prometheus that needs to be cleared?
Thanks Paras. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6bfb92dc-2a18-44d9-8fda-d6f84efba0e7n%40googlegroups.com.
