So initially the pod discovery will give you one target for each container in a pod (see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod), and the only way to get rid of some of those targets is by dropping some of them via relabeling rules with the actions "keep" or "drop". Making two targets look the same in terms of label sets does not combine them into one target. But the only filtering you seem to do is on "__meta_kubernetes_pod_annotation_prometheus_io_scrape", which is an annotation for the whole pod (and you set it on every pod, so nothing gets filtered at all).
If you want to only keep some of the container targets within a pod, you will need to take into account one or more of the container-specific meta labels: - __meta_kubernetes_pod_container_init: true if the container is an InitContainer - __meta_kubernetes_pod_container_name: Name of the container the target address points to. - __meta_kubernetes_pod_container_port_name: Name of the container port. - __meta_kubernetes_pod_container_port_number: Number of the container port. - __meta_kubernetes_pod_container_port_protocol: Protocol of the container port. E.g. the container or port name could make a lot of sense to filter on. Another (maybe better) option, if you already have Kubernetes services defined for just the endpoints you want to scrape in those pods, use the endpoint (instead of pod) discovery in Prometheus to start with a more fitting list of targets to begin with. Btw. as an aside, none of the `prometheus.io/...` annotations have any special meaning in code. They were only used in some example configs as an inspiration for how to base scrape options on Kubernetes annotations, but are not an official or only way of doing things - the only thing that matters is that the names of the annotations fit together with the references to them in your relabeling rules. On Fri, May 15, 2020 at 9:53 AM Scott Mattan <[email protected]> wrote: > please ignore the yaml error -> > ... __address_,} > > the trailing curly brace doesn't actually exist > > 2020年5月15日金曜日 16時50分03秒 UTC+9 Scott Mattan: >> >> I am having a problem with prometheus creating too many targets for a >> specific job, and I cannot understand why I cannot get the correct number >> of targets. >> >> I have the following scrape configuration. >> >> - >> job_name: kubernetes-pods >> kubernetes_sd_configs: >> - {role: pod} >> relabel_configs: >> - {action: keep, regex: true, source_labels: [ >> __meta_kubernetes_pod_annotation_prometheus_io_scrape]} >> - {action: replace, regex: (.+), source_labels: [ >> __meta_kubernetes_pod_annotation_prometheus_io_path], target_label: >> __metrics_path__} >> - { >> source_labels: [__address__, >> __meta_kubernetes_pod_annotation_prometheus_io_port], >> action: replace, >> regex: '([^:]+)(?::\d+)?;(\d+)', >> replacement: '$1:$2', >> target_label: __address__,} >> } >> - {action: labelmap, regex: __meta_kubernetes_pod_label_ >> (.+)} >> - {action: replace, source_labels: [ >> __meta_kubernetes_namespace], target_label: kubernetes_namespace} >> - {action: replace, source_labels: [ >> __meta_kubernetes_pod_name], target_label: kubernetes_pod_name} >> - {regex: '(.*):.+', replacement: '${1}', source_labels: >> [__address__], target_label: instance} >> - {action: replace, source_labels: [ >> __meta_kubernetes_pod_label_cassandra_rook_io_cluster], target_label: >> cluster} >> - {action: replace, source_labels: [ >> __meta_kubernetes_pod_label_cassandra_rook_io_datacenter], target_label: >> dc} >> - {action: replace, source_labels: [ >> __meta_kubernetes_pod_label_cassandra_rook_io_rack], target_label: rack} >> >> This job creates 24 targets from 3 pods with 8 ports each. Each of the >> pods have the following annotations. >> >> >> - prometheus.io/scrape: true >> >> >> - prometheus.io/port: 9180 >> >> >> Due to the above config, shouldn't only one target each for a total of 3 >> targets be created? >> maybe I am understanding the role of `prometheus.io/port` >> <http://prometheus.io/port> ? >> >> If anyone has any information on this I would be grateful >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/2587bcfc-e827-4148-82d0-4d3f79277e1f%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/2587bcfc-e827-4148-82d0-4d3f79277e1f%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5zhUZRctHO8wEMgzgE%3DMaqJ042baaZTa1totot8F_7n8w%40mail.gmail.com.
