Stuart,
Thanks a bunch. That was a huge help.
On Friday, February 5, 2021 at 4:18:09 PM UTC-6 Stuart Clark wrote:
> On 04/02/2021 16:42, Jeff Tippey wrote:
>
> Thanks Stuart,
>
> I have looked around and I can't find it. I'd be grateful if someone
> could take a look at one of these dropped pods labels and my scrape config
> and let me know if you see where it is dropped.
>
> It looks like this bit might be dropping the target:
>
> - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
> action: keep
> target_label: __metrics_path__
> regex: (.+)
>
> It is looking for an annotation called prometheus_io_path which is copied
> to the __metrics_path__ label (which sets the path to scrape).
>
> As this annotation doesn't seem to exist in your example this doesn't
> match, and because the action is keep the target is dropped.
>
> So try adding that annotation with the correct metric path value.
>
> Not working Config
>
> __address__="xxxxxxxx"
> __meta_kubernetes_endpoint_address_target_kind="Pod"
> __meta_kubernetes_endpoint_address_target_name="rcax-ios0-br-1-56z6v"
> __meta_kubernetes_endpoint_port_name="queuemonitor"
> __meta_kubernetes_endpoint_port_protocol="TCP"
> __meta_kubernetes_endpoint_ready="true"
> __meta_kubernetes_endpoints_name="rcax-ios-0"
> __meta_kubernetes_namespace="envb"
>
> __meta_kubernetes_pod_annotation_openshift_io_deployment_config_latest_version="1"
>
>
> __meta_kubernetes_pod_annotation_openshift_io_deployment_config_name="rcax-ios0-br"
>
>
> __meta_kubernetes_pod_annotation_openshift_io_deployment_name="rcax-ios0-br-1"
>
> __meta_kubernetes_pod_annotation_openshift_io_scc="restricted"
> __meta_kubernetes_pod_container_name="queuemonitor"
> __meta_kubernetes_pod_container_port_name=""
> __meta_kubernetes_pod_container_port_number="8001"
> __meta_kubernetes_pod_container_port_protocol="TCP"
> __meta_kubernetes_pod_controller_kind="ReplicationController"
> __meta_kubernetes_pod_controller_name="rcax-ios0-br-1"
> __meta_kubernetes_pod_host_ip="xxxxxxxx"
> __meta_kubernetes_pod_ip="xxxxxxxx"
> __meta_kubernetes_pod_label_component="rcax-ios0-br"
> __meta_kubernetes_pod_label_deployment="rcax-ios0-br-1"
> __meta_kubernetes_pod_label_deploymentconfig="rcax-ios0-br"
> __meta_kubernetes_pod_label_scac="rcax"
> __meta_kubernetes_pod_label_subsystem="rcax-ios0"
> __meta_kubernetes_pod_name="rcax-ios0-br-1-56z6v"
> __meta_kubernetes_pod_node_name="stb-node002"
> __meta_kubernetes_pod_ready="true"
> __meta_kubernetes_pod_uid="356fe668-5a73-11eb-bf8a-0050568584bc"
> __meta_kubernetes_service_annotation_prometheus_io_port="8001"
> __meta_kubernetes_service_annotation_prometheus_io_scheme="http"
> __meta_kubernetes_service_annotation_prometheus_io_scrape="true"
> __meta_kubernetes_service_label_name="rcax-ios0"
> __meta_kubernetes_service_name="rcax-ios-0"
> __metrics_path__="/metrics"
> __scheme__="http"
> job="kubernetes-service-endpoints"
>
>
> Scrape Config
>
> - job_name: 'kubernetes-service-endpoints'
> tls_config:
> ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
> # TODO: this should be per target
> insecure_skip_verify: true
> kubernetes_sd_configs:
> - role: endpoints
> relabel_configs:
> # only scrape infrastructure components
> - source_labels: [__meta_kubernetes_namespace]
> action: keep
> regex:
> 'default|logging|metrics|kube-.+|openshift|openshift-.+|envb|envc'
> # drop infrastructure components managed by other scrape targets
> - source_labels: [__meta_kubernetes_service_name]
> action: drop
> regex: 'prometheus-node-exporter'
> # only those that have requested scraping
> - source_labels:
> [__meta_kubernetes_service_annotation_prometheus_io_scrape]
> action: keep
> regex: true
> - source_labels:
> [__meta_kubernetes_service_annotation_prometheus_io_scheme]
> action: replace
> target_label: __scheme__
> regex: (https?)
> - source_labels:
> [__meta_kubernetes_service_annotation_prometheus_io_path]
> action: keep
> target_label: __metrics_path__
> regex: (.+)
> - source_labels: [__address__,
> __meta_kubernetes_service_annotation_prometheus_io_port]
> action: replace
> target_label: __address__
> regex: (.+)(?::\d+);(\d+)
> replacement: $1:$2
> - action: labelmap
> regex: __meta_kubernetes_service_label_(.+)
> - source_labels: [__meta_kubernetes_namespace]
> action: replace
> target_label: kubernetes_namespace
> - source_labels: [__meta_kubernetes_service_name]
> action: replace
> target_label: kubernetes_name
>
>
>
> On Monday, January 25, 2021 at 4:35:28 PM UTC-6 Stuart Clark wrote:
>
>> On 25/01/2021 22:15, Jeff Tippey wrote:
>>
>> Hi,
>>
>> I have a custom python container I wrote that exports metrics on port
>> 8001 at "/metrics". The setup used to work in Openshift 3.6 with
>> Prometheus. I'm using the Prometheus configuration from github at
>> prometheus
>> github link
>> <https://github.com/openshift/origin/tree/release-3.11/examples/prometheus>.
>>
>>
>> In my deployment yamls for my container, For the list (in deployment
>> yaml) I have annotation
>>
>> annotations:
>> prometheus.io/path: /metrics
>> prometheus.io/port: "8001"
>> prometheus.io/scrape: "true"
>>
>> and the service definition I have
>>
>> annotations:
>> prometheus.io/scrape: "true"
>> prometheus.io/scheme: http
>> prometheus.io/port: "8001"
>>
>> To repeat, this worked in 3.6 configuration. However, when I open up
>> Prometheus this is not a target. Under Targets, none of my service
>> endpoints are showing up. I do see them in the service discovery but they
>> are being dropped. When, I look in the prometheus.yaml, it says that it
>> should scrape anything that is io.scrape = "true". Has anyone experienced
>> this issue or know why it would be dropping my service endpoint scrapes
>> that are defined as io.scrape="true".
>>
>> If not, does anyone know where the reason they are dropped is logged or
>> how to access? I have been looking around and am at a loss right now.
>>
>> Your help is greatly appreciated,
>>
>> If you look through the relabling rules for the job you should be able to
>> see what is going on. Look at the details on the SD page and work out which
>> rule is dropping it. It should then suggest what you need to do to make it
>> work.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/0598e514-fdd5-449c-aa77-5241957c3446n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/prometheus-users/0598e514-fdd5-449c-aa77-5241957c3446n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> Stuart Clark
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/5e67648d-f84f-4843-8bc3-92c7c7807a6cn%40googlegroups.com.
