Thank you for your detailed reply.
Yes, currently we use file discovery. But one important thing maybe I
misssed is that the status of the target might be changed. That is to say,
now the status of the target is the status which we want to monitor, but
after several minis, the target is down, we don't want to monitor the
system anymore.
On Thursday, December 9, 2021 at 6:08:19 PM UTC+8 Brian Candler wrote:
> You could do that - for example, in your inventory system, you expose the
> status of the targets as a prometheus exporter endpoint:
>
> monitoring_active{instance="foo"} 1
> monitoring_active{instance="bar"} 0
>
> Then you scrape this to create a new set of timeseries, and you write your
> alerting expressions to include this values when deciding whether to alert
> or not. These rules become a bit awkward, using one-to-one or many-to-one
> vector matches.
>
> But personally, I would do it the other way around. What I suggest is
> that you generate the list of targets to scrape *from* the inventory system
> in the first place. Then you can either:
> 1. Not include machines with monitoring=no in the list of targets (so they
> don't get scraped at all); or
> 2. Scrape all targets, but add monitoring="no" or monitoring="yes" as a
> target label. You can then use that label in your alerting rules:
>
> # old
> expr: up == 0
>
> # new
> expr: up{monitoring="yes"} == 0
>
> Or even simpler, you can use this target label in your alertmanager rules
> to route the alert to a null endpoint. That is, you still generate alerts
> for all targets, you just don't deliver them if they are labeled with
> monitoring="no".
>
> To do this, you will use the "service discovery" features of prometheus,
> so that your monitoring system *pushes* information to prometheus telling
> it what to scrape and how. The simplest mechanism is the "file_sd"
> mechanism, where you just write the targets into a file.
>
> This is what I do:
> - my inventory is in Netbox
> - I wrote some code <https://github.com/candlerb/netbox-prometheus> to
> read the inventory via Netbox API every 5 minutes and write out the
> prometheus target files (for file_sd_targets)
>
> An example of the generated targets file:
>
> # Auto-generated from Netbox, do not edit as your changes will be
> overwritten!
> - labels:
> netbox_type: device
> targets:
> - nuc1 10.12.255.11
> - nuc2 10.12.255.12
> - storage1 10.12.255.5
>
> You could generate two sets of targets, one with label "monitoring: yes"
> and one with label "monitoring: no"
>
> This has the very nice side benefit that adding a device to your inventory
> will *automatically* add it to prometheus monitoring.
>
> HTH,
>
> Brian.
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/edaad939-fe3e-4fad-accf-b5386083dc43n%40googlegroups.com.
