On 13/12/2021 17:23, mcamm...@gmail.com wrote:
Hi Everyone,
We currently use an external tool to query our list of targets from an
internal database. We then use file based service discovery in
prometheus and provide it the list of targets in a .json file. Works
nicely. This is for the node_exporter job.
- job_name: 'se-linux-node-exporter'
file_sd_configs:
- files:
- 'se-linux-node-exporter.json'
This is 100% of the targets we care about and want to see in our
Grafana dashboards.
We now have a new external search we can run on our internal database
to pull servers based on the service they provide. This will result in
a subset of servers from the above scrape. I would like to add this as
a separate job, like
-job_name: 'se-linux-XYZ-service'
file-_sd_configs:
-files:
- 'se-linux-XYZ-service.json'
My concern is that since this is still for node_exporter port 9100 for
both jobs, I'll get duplicate scrapes ingested.
The end goal is to be able to group servers by service in Grafana -
using the job label. What is the proper way to go about ensuring we
don't have duplicate scrapes/metrics but still maintain this grouping?
I'm hesitant to add custom labeling/relabeling to the original scrape
that has all of the hosts because it could start our graphs over in
Grafana. We have about a two year history (using Victoria Metrics).
It's also possible that any one server can be a part of more than one
service - so we could have duplicates there too.
I'm thinking there's something I can set with relabeling (drop/keep)
or with honor_labels, but I want to be sure I understand what I'm
doing first.
If you have the same endpoints in multiple jobs that will work (as the
job label is different so there aren't duplicates from a time series
perspective) but it isn't very useful, as you'd have duplicates from a
coverage perspective.
If you can have a target being part of multiple services a single label
isn't going to work. Instead you need a label for each service (with a
true/false type value) to be able to encode that, although it really
depends what you are trying to achieve overall.
Normally you'd never try to scrape the same target multiple times, so if
you have something listed in one job you'd make sure it isn't listed in
any others.
The normal way to handle metadata like service, etc. would either be to
add those labels within the job config (often done when that info comes
from places like Kubernetes labels) or by having separate metrics that
you can join into your queries (so something which has something like an
instance label to allow joining but then adds in the service labels).
https://www.robustperception.io/how-to-have-labels-for-machine-roles
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/f041687a-bc2f-a470-2320-aefea4343fb6%40Jahingo.com.
