Hi
I use prometheus to monitor service.
At first, it was simple. I use four labels(team, product, service,
instance) to identify a service, the metric like
```
metric_name{team="t1", product="p1", instance="i1", service="s1", other
business label}
```
I also use the four labels with other labels(such as cpu="xx") to create
alerting rules.
With business development and team adjustment, some team have new child
team and some service deployed to different city, the four labels needs to
be expanded appropriately, may be different scenarios require different
numbers of target labels.
Now I think about two options:
option 1
I will extend target labels for every specific scene.
For every scene, I will store a target labels list in mysql db, different
metric will have diffenert target labels which like
```
metric_name{scene="s1", team="t1", product="p1", instance="i1",
service="s1", ...}
metric_name{scene="s2", team="t2", product="p2", instance="i2",
service="s2", city="c2", ...}
```
disadvantage: I should maintain every scene target labels.
option 2
I will fllow the CMDB which use a tree to organize team, product, service,
city and the relation of them. The tree layers is not fixed, so the target
labels will only have the node id and parent id. Every node have their own
metric. The metric will like
```
metric_name{leatNodeId="n1", pid="p1", ...}
```
disadvantage: aggregating high-layer data is not easy since metric not have
all parent node info.
Is there any other best practices?
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/a517be9f-c02b-4f77-9ddd-fd498d527d1fn%40googlegroups.com.
