Hello everyone ! I've got a question regarding remote_write and federation. I have two levels of prometheus : *level 1* with classic prometheus instances and *level 2* with prometheus federating some metrics from my *level 1*. Both levels are remote_writing to my backend, but I need to find a way not to write the same metrics twice in my backend. >From *level 1*, I remote_write every metrics to the backend, while from the level 2, I need to only remote_write the new ones (calculated on level 2) so I was planning to drop metrics on level 2 based on the federate job name. remote_write: - url: "mybackend" write_relabel_configs: # attempt to drop every federated metric - source_labels: ['job_name'] regex: "federate" action: drop In most cases, this would work, but if a user writes this rule on *level 2* rules: - record: new_metric_on_level_2 expr: sum without(random_label) (counter_from_level_1) new_metric_on_level_2 will inherit the job=federate, thus being dropped at remote_write but it shouldn't as it's a new metric on *level 2 *and needs to be remote_write to my backend. Note : I cannot modify the rule for adding without(random_label,job) even if it solves the problem (because I don't own the rules) Do you have any suggestion on how I could solve this use case ? Thank you so much for reading
-- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20d08e32-100d-48d9-959c-b2fb549d57ecn%40googlegroups.com.
