On 27/07/2023 15:51, Uvais Ibrahim wrote:
Hi Brain,
This is the query that I have used.
sum(scrape_samples_scraped)without(app,app_kubernetes_io_managed_by,clusterName,release,environment,instance,job,k8s_cluster,kubernetes_name,kubernetes_namespace,ou,app_kubernetes_io_component,app_kubernetes_io_name,app_kubernetes_io_version,kustomize_toolkit_fluxcd_io_name,kustomize_toolkit_fluxcd_io_namespace,application,name,role,app_kubernetes_io_instance,app_kubernetes_io_part_of,control_plane,beta_kubernetes_io_arch,beta_kubernetes_io_instance_type,
beta_kubernetes_io_os, failure_domain_beta_kubernetes_io_region,
failure_domain_beta_kubernetes_io_zone,kubernetes_io_arch,
kubernetes_io_hostname, kubernetes_io_os,
node_kubernetes_io_instance_type, nodegroup,
topology_kubernetes_io_region,
topology_kubernetes_io_zone,chart,heritage,revised,transit,component,namespace,
pod_name, pod_template_hash, security_istio_io_tlsMode,
service_istio_io_canonical_name,
service_istio_io_canonical_revision,k8s_app,kubernetes_io_cluster_service,kubernetes_io_name,route_reflector)
Which simply excluded every label but still I am getting a result like
this
{} 7525871918
I'm not sure what you are expecting, as that sounds about right. The
query is adding together all the different variants of the
scrape_samples_scraped metric (removing all the different labels), so if
that is indeed a list of every label the query is going to return a
value without any associated labels.
You want to be instead just graphing the raw scrape_samples_scraped
metric (no sum or without) and see how it varies over time. Is there a
particular job or target which has a huge increase in the graph, or new
series appearing? As to why that might happen it could be many different
reasons, but ideas could include:
* new version of software which increases number of exposed metrics (or
more granular labels)
* bug in software where a label is set to something with high
cardinality (e.g. there is a "path" label from a web app, which means a
potentially infinite cardinality, and you could have had a web scan
producing millions of combinations)
* lots of changes to the targets, such as new instances of software or
high churn of applications restarting
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7423743a-8b29-625d-4472-6aa710cf1179%40Jahingo.com.
