Julius, Thank you! It seems that setting the annotation for `prometheus.io/port` and expecting it to magically work was an error on my part, If I wanted to use that port only I should filter to keep only that port.
That being said, I like the suggestion you gave for service endpoint scraping it really seems like it may be the better option and am going to look into it! 2020年5月15日金曜日 18時25分44秒 UTC+9 Julius Volz: > > So initially the pod discovery will give you one target for each container > in a pod (see > https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod), > > and the only way to get rid of some of those targets is by dropping some of > them via relabeling rules with the actions "keep" or "drop". Making two > targets look the same in terms of label sets does not combine them into one > target. But the only filtering you seem to do is on > "__meta_kubernetes_pod_annotation_prometheus_io_scrape", which is an > annotation for the whole pod (and you set it on every pod, so nothing gets > filtered at all). > > If you want to only keep some of the container targets within a pod, you > will need to take into account one or more of the container-specific meta > labels: > > - __meta_kubernetes_pod_container_init: true if the container is an > InitContainer > - __meta_kubernetes_pod_container_name: Name of the container the target > address points to. > - __meta_kubernetes_pod_container_port_name: Name of the container port. > - __meta_kubernetes_pod_container_port_number: Number of the container > port. > - __meta_kubernetes_pod_container_port_protocol: Protocol of the container > port. > > E.g. the container or port name could make a lot of sense to filter on. > > Another (maybe better) option, if you already have Kubernetes services > defined for just the endpoints you want to scrape in those pods, use the > endpoint (instead of pod) discovery in Prometheus to start with a more > fitting list of targets to begin with. > > Btw. as an aside, none of the `prometheus.io/...` > <http://prometheus.io/...> annotations have any special meaning in code. > They were only used in some example configs as an inspiration for how to > base scrape options on Kubernetes annotations, but are not an official or > only way of doing things - the only thing that matters is that the names of > the annotations fit together with the references to them in your relabeling > rules. > > On Fri, May 15, 2020 at 9:53 AM Scott Mattan <[email protected] > <javascript:>> wrote: > >> please ignore the yaml error -> >> ... __address_,} >> >> the trailing curly brace doesn't actually exist >> >> 2020年5月15日金曜日 16時50分03秒 UTC+9 Scott Mattan: >>> >>> I am having a problem with prometheus creating too many targets for a >>> specific job, and I cannot understand why I cannot get the correct number >>> of targets. >>> >>> I have the following scrape configuration. >>> >>> - >>> job_name: kubernetes-pods >>> kubernetes_sd_configs: >>> - {role: pod} >>> relabel_configs: >>> - {action: keep, regex: true, source_labels: [ >>> __meta_kubernetes_pod_annotation_prometheus_io_scrape]} >>> - {action: replace, regex: (.+), source_labels: [ >>> __meta_kubernetes_pod_annotation_prometheus_io_path], target_label: >>> __metrics_path__} >>> - { >>> source_labels: [__address__, >>> __meta_kubernetes_pod_annotation_prometheus_io_port], >>> action: replace, >>> regex: '([^:]+)(?::\d+)?;(\d+)', >>> replacement: '$1:$2', >>> target_label: __address__,} >>> } >>> - {action: labelmap, regex: __meta_kubernetes_pod_label_ >>> (.+)} >>> - {action: replace, source_labels: [ >>> __meta_kubernetes_namespace], target_label: kubernetes_namespace} >>> - {action: replace, source_labels: [ >>> __meta_kubernetes_pod_name], target_label: kubernetes_pod_name} >>> - {regex: '(.*):.+', replacement: '${1}', source_labels: >>> [__address__], target_label: instance} >>> - {action: replace, source_labels: [ >>> __meta_kubernetes_pod_label_cassandra_rook_io_cluster], target_label: >>> cluster} >>> - {action: replace, source_labels: [ >>> __meta_kubernetes_pod_label_cassandra_rook_io_datacenter], target_label: >>> dc} >>> - {action: replace, source_labels: [ >>> __meta_kubernetes_pod_label_cassandra_rook_io_rack], target_label: rack} >>> >>> This job creates 24 targets from 3 pods with 8 ports each. Each of the >>> pods have the following annotations. >>> >>> >>> - prometheus.io/scrape: true >>> >>> >>> - prometheus.io/port: 9180 >>> >>> >>> Due to the above config, shouldn't only one target each for a total of 3 >>> targets be created? >>> maybe I am understanding the role of `prometheus.io/port` >>> <http://prometheus.io/port> ? >>> >>> If anyone has any information on this I would be grateful >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/2587bcfc-e827-4148-82d0-4d3f79277e1f%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/2587bcfc-e827-4148-82d0-4d3f79277e1f%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1889c4cf-9a68-4013-8381-9d1205808b3a%40googlegroups.com.
