[prometheus-users] Prometheus throwing unexpected EOF for targets
Hi All I have deployed prometheus with prometheus operator.Our prometheus was running on 2.15.2 and recently we have upgraded prometheus to 2.36.1 along with prometheus operator to 0.57.0. After upgrade we are getting *unexpected EOF *for targets.My issue looks simillar to the reported one https://github.com/prometheus/prometheus/issues/10213. I dont see any option to disable http2 with operator 0.57.0. How can I debug this and how can I disable http2 Thanks Kirti -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ac353cc5-dec5-4835-9756-2e3d10178de7n%40googlegroups.com.
[prometheus-users] Re: Prometheus azure SD throttling Azure API limit
Can someone check on this please On Thursday, 4 February 2021 at 19:44:08 UTC+5:30 Kirti Ranjan Parida wrote: > Hello everyone > I have a question regarding azure_sd_configs. My job is discovering 1600 > targets(refresh_interval=5mins) out of which only 88 are active targets and > others are dropped by prometheus relabelling. However We are hitting > compute/network API limit just because of this and other application are > not able to make API calls > >- Does Prometheus makes describe call to all the instances/NIC’s that >it discovers ? >- Any other way to restrict the number of API calls >since azure_sd_configs does not support filter: ? > > > Any other workaround ? > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7dd32557-0bb4-4a0a-8b71-5e064a30cc64n%40googlegroups.com.
[prometheus-users] Prometheus azure SD throttling Azure API limit
Hello everyone I have a question regarding azure_sd_configs. My job is discovering 1600 targets(refresh_interval=5mins) out of which only 88 are active targets and others are dropped by prometheus relabelling. However We are hitting compute/network API limit just because of this and other application are not able to make API calls - Does Prometheus makes describe call to all the instances/NIC’s that it discovers ? - Any other way to restrict the number of API calls since azure_sd_configs does not support filter: ? Any other workaround ? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f1e196d3-11ac-4ddb-9a2e-13524e3aac6dn%40googlegroups.com.
[prometheus-users] How to monitor PODS that are in error state
How to monitor PODS that are in error state -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1ec2f1ba-ae8a-46c4-8224-352e6ea08131n%40googlegroups.com.
[prometheus-users] How to use azure_sd_configs to scrape azure VM's and VMSS's
I am trying to scrape VM's and VMSS's in azure using prometheus.For discovery I am using azure_sd_configs.Below is my configuration - job_name: 'node_exporter' azure_sd_configs: - authentication_method: ManagedIdentity subscription_id: refresh_interval: 5m port: 9100 Do I have to mention anything ? I am getting below error level=warn ts=2020-06-30T08:13:55.895Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="/app/discovery/kubernetes/kubernetes.go:261: watch of *v1.Endpoints ended with: too old resource version: 1291352 (1292845)" level=error ts=2020-06-30T08:15:28.938Z caller=refresh.go:98 component="discovery manager scrape" discovery=azure msg="Unable to refresh target groups" err="could not get virtual machines: could not list virtual machines: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions//providers/Microsoft.Compute/virtualMachines?api-version=2018-10-01: StatusCode=404 -- Original Error: adal: Refresh request failed. Status Code = '404'. Response body: getting assigned identities for pod in CREATED state failed after 16 attempts, retry duration [5]s. Error: \n" Any help ? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/eb6678a0-aac2-4e0f-9746-1b391185598do%40googlegroups.com.
[prometheus-users] Re: getting certificate issue while connecting to k8s server
Did anyone got resolution to this ? Even I am also facing the same problem On Wednesday, 3 October 2018 12:30:27 UTC+5:30, mattia...@eyeonid.com wrote: > > Did you find a solution to this? > > On Tuesday, 10 April 2018 10:09:26 UTC+2, an...@a10networks.com wrote: >> >> I am setting up k8s monitoring from outside (different prometheus server >> . but i can access private IP of k8s server). I generated the certificates >> (ref is https://rootsquash.com/2016/05/10/securing-the-kubernetes-api/) >> for my Prometheus server on k8s and copied to Prometheus server. also made >> entry in prometheus.yml file as below. >> >> - job_name: 'k8s_testing' >> scrape_interval: 15s >> honor_labels: true >> kubernetes_sd_configs: >> - role: node >> api_server: https://172.30.0.89:6443 >> tls_config: >> cert_file: >> /etc/prometheus/k8s_certs/k8scl2/ip-172-30-0-90.ec2.internal.crt >> key_file: >> /etc/prometheus/k8s_certs/k8scl2/ip-172-30-0-90.ec2.internal.key >> ca_file: /etc/prometheus/k8s_certs/k8scl2/ca.pem >> insecure_skip_verify: true >> relabel_configs: >> - action: labelmap >> regex: __meta_kubernetes_node_label_(.+) >> - source_labels: [__address__] >> regex: '(.*):10250' >> replacement: '${1}:10255' >> target_label: __address__ >> >> after restarting prometheus service. I am getting below error. >> >> Apr 10 07:41:58 ip-172-30-0-11 prometheus: level=error >> ts=2018-04-10T07:41:58.46774488Z caller=main.go:221 >> component=k8s_client_runtime err=" >> github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:354: >> Failed to list *v1.Node: Get >> https://172.30.0.89:6443/api/v1/nodes?resourceVersion=0: x509: >> certificate signed by unknown authority" >> >> when i cross checked Issuer is kubernetes in both (apiserver's pem file >> and prometheus server's certificate). if issuer is same then why it is >> saying certificate signed by unknown authority. >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/78f801bc-a67c-4cc9-988d-7d6f36456cf1o%40googlegroups.com.