A similar question was asked here but received no answer <https://groups.google.com/g/prometheus-users/c/pDXOohGs2iQ/m/5T-1F_OHAgAJ>, was not sure if I should reply to that thread or create a new one.
*Goal*: Users create prometheus alert rules as necessary, these shall be routed to alertmanager routes, including webhook configs (because Microsoft Teams is a receiver). Users should not need to make changes to default config files (alert manager) because these are version controlled. The webhook url has to come / be constructed from alert labels (see below why). *Problem*: It does not seem possible to define a webhook, *that does not have a constant URL*, because the url property in webhook_config is not a <tmpl_string> , no template expansion is possible. Why? Would the prometheus team accept a change to make webhook url a <tmpl_string>? *Why this is needed: * Our default alertmanager config file is committed to git. It is not feasible to change every alertmanager config yaml on every server instance (on-prem), users should not change this file, ever, if possible. When rolling out updates to the config, user changes would be overwritten and lost. All alertmanagers should run using known default configurations / routes. which have to be flexible enough to handle all use cases. Example: Alarms concern certain teams of certain apps / projects / customers (-> msteams / slack channels). If n-amount of webhook urls for n-amount of apps are pre-defined alertmanager config, and a new app is deployed (new webhook target), the version controlled alertmanager config file must be updated. This is troublesome. If a unique channel exists for custom foo, the webhook url has to be added in any case during installation of alertmanager. But users are able to provide new alert rules for prometheus, since those can be reloaded dynamically (from rule files) without changing prometheus server config yaml (which is version controlled too). So providing the webhook url as a alert rule label value would allow dynamic routing. In contrast, this seems very easily possible using email routing. In this blog post <https://www.robustperception.io/using-labels-to-direct-email-notifications> by Brian Brazil it is demonstrated, how to use email_configs: - to: "{{ .GroupLabels.email_to }}" to achive dynamic routing, because the "to" property is a <tmpl_string> Additional info: I was planning to use this proxy: https://github.com/prometheus-msteams/prometheus-msteams which has several configuration possibilities, even one called dynamic <https://github.com/prometheus-msteams/prometheus-msteams#dynamic-uri-handler-_dynamicwebhook>, but the problem remains: Some URL has to come from alertmanager. Currently my only remaining idea (besides forking and add this feature to prometheus-msteams) is to write yet another simple http server between alertmanager and prometheus-msteams, read the alert JSON properties, construct the webhook URL and forward the request accordingly to prometheus-msteams. This seems overkill. *Am I overlooking something? Please advise. * Thank you. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f359428-9fde-4a4a-83d7-bec8bea02263n%40googlegroups.com.
