Re: [prometheus-users] Alert on no alerts?

2021-02-02 Thread Ben Kochie
You can self-host your own heartbeat service with healthchecks.io https://github.com/healthchecks/healthchecks On Tue, Feb 2, 2021 at 11:47 PM Julius Volz wrote: > No, Alertmanager cannot do this by itself. In any case it's a good idea to > have a third party for this because the whole idea is

Re: [prometheus-users] Alert on no alerts?

2021-02-02 Thread Julius Volz
No, Alertmanager cannot do this by itself. In any case it's a good idea to have a third party for this because the whole idea is that it should be as independent from your infrastructure as possible, so you even still get an alert even if you Alertmanager has a problem too. On Tue, Feb 2, 2021 at

Re: [prometheus-users] Alert on no alerts?

2021-02-02 Thread Peter S
Thanks! Really amazing. Is there a way to do this without using third-party service? I'm hoping AlertManager can detect that the heatbeat alert has stopped and fire off an alert? Thanks! On Sunday, January 31, 2021 at 3:56:23 AM UTC-5 juliu...@promlabs.com wrote: > Yep! What you want is an

Re: [prometheus-users] Alert on no alerts?

2021-01-31 Thread Julius Volz
Yep! What you want is an alerting heartbeat: https://www.youtube.com/watch?v=RsigFUMUHZ0 So an external service like https://deadmanssnitch.com/ that sends you a notification when it does *not* receive a message in a given time. If you are using the Prometheus Operator to deploy Prometheus to

[prometheus-users] Alert on no alerts?

2021-01-30 Thread 'ping...@hioscar.com' via Prometheus Users
Is there a way for AlertManager to fire an alert no receiving no alerts in the last like 10 mins? We recently experienced an issue when prometheus got misconfigured and stopped sending alerts, including alerts on itself. So, to detect this scenario, we cannot rely on prometheus alert rules. I