Hi, On 3/15/20 10:07 AM, Yagyansh S. Kumar wrote: > Thanks for the quick response. I appreciate your advice and I know that > instance label shouldn't contain the port number and that should be the > ideal way forward, but now my setup is huge. I'll have to change things > over all my dashboards and I am monitoring close to 3000 servers. > Eventually, I am going to change the instance label altogether but > currently I need a short term solution to remove the port from alert > annotations. Is it at all possible?
Some ideas come to mind. I would call them all workarounds, as they can become rather ugly maintenance-wise. 1) Have another meta metric which provides an instance="a:9100" to "instance_ip" mapping. This could be generated via a recording rule in combination with label_replace(), e.g.: - record: instance_to_ip_mapping expr: count by(instance, instance_ip) (label_replace(up, "instance_ip", "$1", "instance", "(.*):\d+")) https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace You could then mix in this metric into the relevant alerts. 2) You could attach such an instance_ip label to some or all of your metrics using relabeling. This will likely cause a larger overhead though. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs 3) You could run an additional query in your alerts, e.g. dashboard: "DashbordLink?orgId=1&var-node={{ printf "label_replace(up{instance='%s'}, 'instance_ip', '$1', 'instance', '(.*):\d+'" .Labels.instance | query | first | label 'instance_ip' }}&var-name={{ $labels.nodename }}&fullscreen&panelId=174" 4) You could try to offload this mapping to Grafana, i.e. make your dashboard work with just a name selected and let Grafana do the mapping. Not sure, if/how this is possible, might also depend on your dashboard and the variables. All in all, I would still follow Brian's suggestion of getting the instance= label right in the first place. It is possible -- we went through the same process at some time. ;) As always, all untested examples -- it's likely I've got some syntax/quoting/escaping wrong there. If you end up using one of those examples, please report back with the final solution, so that others can also benefit from this. :) Kind regards, Christian -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5daa9314-2efa-559f-aeb6-bc40f06bf4b2%40hoffmann-christian.info.
