No, Alertmanager cannot do this by itself. In any case it's a good idea to have a third party for this because the whole idea is that it should be as independent from your infrastructure as possible, so you even still get an alert even if you Alertmanager has a problem too.
On Tue, Feb 2, 2021 at 11:11 PM Peter S <shanp...@gmail.com> wrote: > Thanks! Really amazing. Is there a way to do this without using > third-party service? I'm hoping AlertManager can detect that the heatbeat > alert has stopped and fire off an alert? Thanks! > > On Sunday, January 31, 2021 at 3:56:23 AM UTC-5 juliu...@promlabs.com > wrote: > >> Yep! What you want is an alerting heartbeat: >> https://www.youtube.com/watch?v=RsigFUMUHZ0 >> >> So an external service like https://deadmanssnitch.com/ that sends you a >> notification when it does *not* receive a message in a given time. >> >> If you are using the Prometheus Operator to deploy Prometheus to >> Kubernetes using kube-prometheus, you already get a default alerting rule >> for that: >> https://github.com/prometheus-operator/kube-prometheus/blob/8588e30bd02c60ca0ed75f450014a2b1c1e8ff5a/manifests/kube-prometheus-prometheusRule.yaml#L23-L33 >> >> On Sun, Jan 31, 2021 at 3:50 AM 'pin...@hioscar.com' via Prometheus >> Users <promethe...@googlegroups.com> wrote: >> >>> Is there a way for AlertManager to fire an alert no receiving no alerts >>> in the last like 10 mins? We recently experienced an issue when prometheus >>> got misconfigured and stopped sending alerts, including alerts on itself. >>> So, to detect this scenario, we cannot rely on prometheus alert rules. I >>> hope AlertManager can fire an alert on its own when receiving no alerts for >>> some period of time. Please advise. Thanks >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-use...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/e8fee1c3-6e37-4b1c-8042-0fa066b6052en%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/e8fee1c3-6e37-4b1c-8042-0fa066b6052en%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Julius Volz >> PromLabs - promlabs.com >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c3956927-c3ca-4232-a7f9-bb64c99c9fcen%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/c3956927-c3ca-4232-a7f9-bb64c99c9fcen%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5zRF%2BHrncBxi5XQrQNsdDDFMx3z1Lb7RSxpJFCavmx43g%40mail.gmail.com.
