Re: [prometheus-developers] Is this alerting architecture crazy?

2021-11-20 Thread Stuart Clark
It sounds like you are planning on creating a fairly complex system that duplicates a reasonable amount of what Alertmanager already does. I'm presuming your diagram is a simplification and that the application is itself a cluster, so each instance would be querying each instance of

Re: [prometheus-developers] Is this alerting architecture crazy?

2021-11-20 Thread Tony Di Nucci
Yes, the diagram is a bit of a simplification but not hugely. There may be multiple instances of AlertRouter however they will share a database. Most likely things will be kept simple (at least initially) where each instance holds no state of its own. Each active alert in the DB will be

[prometheus-developers] É NAZIST卍ASSASSINO #ROBERTOJONGHILAVARINI (PURE PEDOFILO E PARTE DI NDRANGHETA) DI CRIMINALISSIMO ISTITUTO GANASSINI DI RICERCHE BIOMEDICHE E CRIMINALISSIMO MOVIMENTO #FAREFRON

2021-11-20 Thread MICHELE CALZOLARI. ASSOSIM, HI-MTF SIM E SARDINE
É NAZIST卍ASSASSINO #ROBERTOJONGHILAVARINI (PURE PEDOFILO E PARTE DI NDRANGHETA) DI CRIMINALISSIMO ISTITUTO GANASSINI DI RICERCHE BIOMEDICHE E CRIMINALISSIMO MOVIMENTO #FAREFRONTE FARE FRONTE! IL FIGLIO DI PUTTANA OMICIDA ROBERTO JONGHI LAVARINI...SE LA FA MOLTO, A LIVELLO DI RICICLAGGIO DI

Re: [prometheus-developers] Is this alerting architecture crazy?

2021-11-20 Thread Ben Kochie
What gives you the impression that the Alertmanager is "best effort"? The alertmanager provides a reasonably robust HA solution (gossip clustering). The only thing best-effort here is actually deduplication. The Alertmanager design is "at least once" delivery, so it's robust against network

Re: [prometheus-developers] Is this alerting architecture crazy?

2021-11-20 Thread Ben Kochie
Also, the alertmanager does have an "even store", it's a shared state between all instances. If you're interested in changing some of the behavior of the retry mechanisms or how this works, feel free to open specific issues. You don't need to build an entirely new system, we can add new features

[prometheus-developers] Is this alerting architecture crazy?

2021-11-20 Thread Tony Di Nucci
Cross-posted from https://discuss.prometheus.io/t/is-this-alerting-architecture-crazy/610 In relation to alerting, I’m looking for a way to get strong alert delivery guarantees (and if delivery is not possible I want to know about it quickly). Unless I’m mistaken AlertManager only offers

Re: [prometheus-developers] Is this alerting architecture crazy?

2021-11-20 Thread Tony Di Nucci
Thanks for the feedback. > What gives you the impression that the Alertmanager is "best effort"? Sorry, best-effort probably wasn't the right term to use. I am aware of there being retries however these could still all fail and I'm thinking I wouldn't be made aware of the issue for potentially

Re: [prometheus-developers] Is this alerting architecture crazy?

2021-11-20 Thread Tony Di Nucci
There are other things I need to do as well, alert enrichment, complex routing, etc. which means that I think some additional system is needed between AlertManager and the final destination in any case. The main question in my mind is really; are there reasons why I should prefer to have