On 18/12/2020 02:54, Allenzh li wrote:
Hi
I use prometheus to monitor service.
At first, it was simple. I use four labels(team, product, service,
instance) to identify a service, the metric like
```
metric_name{team="t1", product="p1", instance="i1", service="s1",
other business label}
```
I also use the four labels with other labels(such as cpu="xx") to
create alerting rules.
With business development and team adjustment, some team have new
child team and some service deployed to different city, the four
labels needs to be expanded appropriately, may be different scenarios
require different numbers of target labels.
Now I think about two options:
option 1
I will extend target labels for every specific scene.
For every scene, I will store a target labels list in mysql db,
different metric will have diffenert target labels which like
```
metric_name{scene="s1", team="t1", product="p1", instance="i1",
service="s1", ...}
metric_name{scene="s2", team="t2", product="p2", instance="i2",
service="s2", city="c2", ...}
```
disadvantage: I should maintain every scene target labels.
option 2
I will fllow the CMDB which use a tree to organize team, product,
service, city and the relation of them. The tree layers is not fixed,
so the target labels will only have the node id and parent id. Every
node have their own metric. The metric will like
```
metric_name{leatNodeId="n1", pid="p1", ...}
```
disadvantage: aggregating high-layer data is not easy since metric not
have all parent node info.
Is there any other best practices?
Rather than attaching the team, etc labels to every metrics, it can work
well to instead have separate metrics with that information, similar to
machine role metrics as described here:
https://www.robustperception.io/how-to-have-labels-for-machine-roles
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ff4e0911-c66c-5403-d0a7-393cfeb415bc%40Jahingo.com.
