Hi all,
In our environment for monitoring about 200 micro-services, we use
Prometheus & Grafana.
In one application to another, developers used different different strings
as the namespace component.
i.e. we have used Prometheus keys like
"booking_engine_driver_eta_location_service_outboundcall_latency_microseconds_count"
to count the latency from "BookingEngine.Driver-ETA" to "Location-Service"
In this "Booking Engine" is the "Service Group" and "Driver-ETA" is the
service and "Location-Service" is the outbound service
In monitoring its a must to monitor "Inbound Request Rates by Endpoint",
"Inbound Request Error Rates by Endpoint", "Processing Latency by
Endpoint", "Outbound Request Rates by Endpoint", "Outbound Request Rates by
Endpoint", "Outbound Request Error Rates by Endpoint" for API based
requests.
We can monitor all the services with about 3 dashboards "Inbound Service
Monitor Rates", "Outbound Service Monitor Rates", "Processing Latencies" we
know the Prometheus keys used.
So we wanted to standardize the Prometheus Keys as the following
- We use namespace to define the "Development Team"
- Application Name will be a label in the key - i.e. label will be "app"
- Endpoint also will be a label in the key
- Error will be a label in the key
So the previous key with labels will be changed to
"outboundcall_latency_microseconds_count{app="booking_engine_driver_eta_location_service"}"
Doing this we can automate most of the things related Dashboarding and
Alerting.
By doing this about 200 time series-es will be grouped into about 4 groups
and hence 200 time series into 4 time series.
Doing so, will there be a big hit for Prometheus performance?
As Prometheus sounds, it can handle millions of time series but even though
we did so, there will be about 50,000 time series. But in one time series,
there are data for about 200 services. To get the latency count expression
would be
"outboundcall_latency_microseconds_count{app="booking_engine_driver_eta_location_service"}".
Is it advisable to do like this?
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/fbea433a-9d04-4e0c-95c5-820b75b37facn%40googlegroups.com.
