Re: [prometheus-users] kubernetes SD config for Ingressroute Traefik
Hi,
The metrics exposed would look like below.
traefik_entrypoint_request_duration_seconds_bucket{code="200",entrypoint="https",method="GET",protocol="http",le="0.1"}
2
traefik_entrypoint_request_duration_seconds_bucket{code="200",entrypoint="https",method="GET",protocol="http",le="0.3"}
50
traefik_entrypoint_request_duration_seconds_bucket{code="200",entrypoint="https",method="GET",protocol="http",le="1.2"}
51
traefik_entrypoint_request_duration_seconds_bucket{code="200",entrypoint="https",method="GET",protocol="http",le="5"}
55
traefik_entrypoint_request_duration_seconds_bucket{code="200",entrypoint="https",method="GET",protocol="http",le="+Inf"}
55
These are exposed by traefik and not by the service.
Thanks
Eswar
On Saturday, 8 August 2020 14:27:55 UTC+2, Matthias Rampke wrote:
>
> Can you explain more what "metrics exposed by ingressroute" looks like?
> This only configures traefik; are the metrics you are looking for exposed
> by traefik or by the service that the ingressroute refers to?
>
> /MR
>
> On Fri, Aug 7, 2020, 09:27 Ishvar B >
> wrote:
>
>> Hi,
>>
>> I am trying to use auto discovery of metrics endpoints for my k8s
>> cluster. While I am able to scrape pod and service level metrics, i have
>> metrics exposed by ingressroute(traefik). So if anybody knows how to use
>> auto discovery with traefik, it would of great help.
>>
>> Thanks
>> Eswar
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to promethe...@googlegroups.com .
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/4fddc1e1-98d5-4036-9cad-bd2123bec199o%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/4fddc1e1-98d5-4036-9cad-bd2123bec199o%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/bd62399c-057d-4ebb-9418-848d3dee85e4o%40googlegroups.com.
[prometheus-users] kubernetes SD config for Ingressroute Traefik
Hi, I am trying to use auto discovery of metrics endpoints for my k8s cluster. While I am able to scrape pod and service level metrics, i have metrics exposed by ingressroute(traefik). So if anybody knows how to use auto discovery with traefik, it would of great help. Thanks Eswar -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4fddc1e1-98d5-4036-9cad-bd2123bec199o%40googlegroups.com.
[prometheus-users] Re: scrape metrics from k8s pod using basic auth
Hi Brain, Thanks for your reply. I already had config to scrape using basic auth. but my questions , can these be used in conjunction. Prometheus first using kubernetes SD config will identify the targets to be scrapped and then using basic auth to scrape individual endpoints identified in the first step. Thanks Eswar On Thursday, 23 July 2020 16:55:04 UTC+2, Brian Candler wrote: > > > Is it possible by prom to scrape these apps behind basic auth? I could > not find any documentation > > Yes. Set the basic_auth at the "scrape_config" level, and it will be used > when scraping the target (as opposed to basic_auth under the kubernetes > SD config, which would be used for talking to the kubernetes API) > > > https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5733e528-77be-4aa9-b832-451a75fbfdd3o%40googlegroups.com.
[prometheus-users] scrape metrics from k8s pod using basic auth
Hi All, I have got apps running inside k8s cluster. I have my prom set up outside of the cluster and below is the config to scrape pod level metrics. This config is picked from prometheus.io kubernetes SD config. We are planning to deploy some apps to k8s but these apps will have metrics exposed behind basic auth. Is it possible by prom to scrape these apps behind basic auth? I could not find any documentation and this looks quite confusing too. - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod api_server: "$API_SERVER_IP" tls_config: insecure_skip_verify: true basic_auth: username: admin password: "$K3S_ADMIN_PASSWORD" scheme: https tls_config: insecure_skip_verify: true basic_auth: username: admin password: "$K3S_ADMIN_PASSWORD" relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace regex: () target_label: __meta_kubernetes_pod_annotation_prometheus_io_path replacement: /metrics - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_annotation_io_port, __meta_kubernetes_pod_annotation_prometheus_io_path] target_label: __metrics_path__ regex: (.+);(.+);(.+);(.+) replacement: /api/v1/namespaces/$1/pods/$2:$3/proxy$4 - target_label: __address__ replacement: $API_SERVER_IP - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name - source_labels: [__meta_kubernetes_pod_node_name] action: replace target_label: instance Thanks Eswar -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ab2ad422-d18d-41f1-84c7-125edb0ac18eo%40googlegroups.com.
[prometheus-users] need assistance with scrapping metrics from k3s cluster
HI, I have got a k3s cluster (not k8s). The metrics of my cluster is exposing api metrics. I don't have metrics server installed so right now what ever the metrics that i am getting is from api server. I want to know or have example scrape config for these metrics. The prometheus documentation page has only got k8s sd config which unfortunately, i cannot use in this case. Thanks Eswar -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7536960b-1f0b-47da-8cc9-12638c9d0979o%40googlegroups.com.
[prometheus-users] need help in scraping oracle metrics using oracle exporter.
Hi All, I am trying to scrape oracle metrics in to my prometheus. I am trying to use a docker container as I don't want to know run the exporter on all the vms running oracle. I have zeroed in https://github.com/aoliynik/oracle-exporter. But there is very less documentation on how to run this? struggling to get this up and running. Has anyone used same/similar exporter and how to go around with the configuration. Thanks Eswar -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e3182f25-6afd-42d2-ae17-d9ccdbaf49fdo%40googlegroups.com.
[prometheus-users] need sample config for alertmanager
Hi,
I have got prometheus and alert manager set up and running. I am using
pagerduty as receiver from alertmanager. Now, i want to create separate
services in pagerduty for separate applications. How do i configure
alertmanager to send these alerts to the service based on the application
name?
How does default-receiver work here? Also, application is a label in my
prometheus.yml config. Am i correct in saying that the application comes
from prometheus.yml config?
global:
resolve_timeout: 1m
route:
receiver: default-receiver
routes:
- receiver: teamA
match:
application: appA
- receiver: teamB
match:
application: appB
receivers:
- name: default-receiver
- name: teamA
pagerduty_configs:
- send_resolved: true
routing_key: ${PAGERDUTY_KEY_APPA}
description: '{{ template "pagerduty.default.description" .}}'
severity: '{{ .CommonLabels.severity }}'
details:
summary: |-
{{ range .Alerts }}{{ .Annotations.summary }}
{{ end }}
severity: '{{ .CommonLabels.severity }}'
status: '{{ .Status }}'
- name: teamB
- send_resolved: true
routing_key: ${PAGERDUTY_KEY_APPB}
description: '{{ template "pagerduty.default.description" .}}'
severity: '{{ .CommonLabels.severity }}'
details:
summary: |-
{{ range .Alerts }}{{ .Annotations.summary }}
{{ end }}
severity: '{{ .CommonLabels.severity }}'
status: '{{ .Status }}'
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ab122ff3-6747-4fb2-bcb7-7b258a284a76o%40googlegroups.com.
Re: [prometheus-users] Prometheus predict linear query needs to exclude a metrics for a certain host
Hi Julius,
Thanks for your reply.
I could not completely understand. I have read about recording rules but in
this context what should come in the place of expr: 1 , should that be
node_filesystem_free_bytes{fstype!~"tmpfs",job!~"example.*",
mountpoint!~"/var/cache/fscache"}?
Thanks
Eswar
On Friday, 15 May 2020 12:01:29 UTC+2, Julius Volz wrote:
>
> There's two options:
>
> - You can set the --collector.filesystem.ignored-mount-points (default
> "^/(dev|proc|sys|var/lib/docker/.+)($|/)") on the specific hosts to not
> even export metrics about that mountpoint, if you never need those metrics
> at all.
>
> - If that's not an option, you could have static recording rules that
> create time series with the instance+job labels that you would want to
> exclude, like:
>
> groups:
> - name: excludes
> rules:
> - record: excluded_instance_mountpoints
> expr: 1
> labels:
>instance:
>mountpoint:
> - record: excluded_instance_mountpoints
> expr: 1
> labels:
>instance:
>mountpoint:
>
> ...and then you could use those recorded time series to remove alerts for
> those exceptions:
>
> predict_linear(node_filesystem_free_bytes{fstype!~"tmpfs"}[1h], 4 *
> 3600) < 0
> unless on(instance, mountpoint)
> excluded_instance_mountpoints
>
> (you could change "instance" to "job" in the example if indeed your
> exceptions are grouped by "job" in your case)
>
> On Fri, May 15, 2020 at 11:30 AM Ishvar B >
> wrote:
>
>> Hi,
>>
>> I need help for the below issue.
>>
>> I have the below query regarding which as per the query definition is
>> working fine. But for certain host (say example*), I want to exclude
>> /var/cache/fscache metric from alert rule. I am little confused as either
>> it excludes the metric for all the hosts or collects it for all the hosts.
>>
>> query: predict_linear(node_filesystem_free_bytes{fstype!~"tmpfs"}[1h], 4
>> * 3600) < 0 - This collects all metrics for all the hosts
>>
>> updated query:
>> predict_linear(node_filesystem_free_bytes{fstype!~"tmpfs",job!~"example.*",mountpoint!~"/var/cache/fscache"}[1h],
>>
>> 4 * 3600) < 0 - This excludes both the host example* and the mountpoint
>> /var/cache/fscache. I want this to alert for all the metrics for all the
>> hosts but for example* host don't alert for mounpoint /var/cache/fscache.
>>
>> Any help is highly appreciated.
>>
>> Thanks
>> Eswar
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to promethe...@googlegroups.com .
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/253f05a4-5428-4b00-af77-0ca93f2c93c8%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/253f05a4-5428-4b00-af77-0ca93f2c93c8%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
> Julius Volz
> PromLabs - promlabs.com
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/f67755a9-397f-4595-b87b-97d2e94189e4%40googlegroups.com.
[prometheus-users] Prometheus predict linear query needs to exclude a metrics for a certain host
Hi,
I need help for the below issue.
I have the below query regarding which as per the query definition is
working fine. But for certain host (say example*), I want to exclude
/var/cache/fscache metric from alert rule. I am little confused as either
it excludes the metric for all the hosts or collects it for all the hosts.
query: predict_linear(node_filesystem_free_bytes{fstype!~"tmpfs"}[1h], 4 *
3600) < 0 - This collects all metrics for all the hosts
updated query:
predict_linear(node_filesystem_free_bytes{fstype!~"tmpfs",job!~"example.*",mountpoint!~"/var/cache/fscache"}[1h],
4 * 3600) < 0 - This excludes both the host example* and the mountpoint
/var/cache/fscache. I want this to alert for all the metrics for all the
hosts but for example* host don't alert for mounpoint /var/cache/fscache.
Any help is highly appreciated.
Thanks
Eswar
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/253f05a4-5428-4b00-af77-0ca93f2c93c8%40googlegroups.com.
[prometheus-users] Re: Too many scrape configs on prometheus config file
Hi Brain, Thanks for you reply. I was able to figure out yesterday. Is there a way to give basic auth details in the yaml file. I have been trying to add basic auth and prometheus is not picking them up. sample config. - targets: ['**.**.**'] scheme: "https" basic_auth: username: "" password: "" labels: job: '**' Thanks Eswar On Tuesday, 31 March 2020 16:59:52 UTC+2, Ishvar B wrote: > > Hi, > > I have a prometheus set up which has too many (roughly 50 static configs) > and this number would increase as we have lot of vms still be scraped for > prometheus. The file is becoming huge , especially when some applications > require basic auth. is there any best practice to maintain the config file > as with such huge file some times changes would cause accidental errors. > > Thanks > Eswar > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b0a038f1-29a5-4c77-8f58-5630969c2abb%40googlegroups.com.
Re: [prometheus-users] Too many scrape configs on prometheus config file
Hi, Thanks. I have used file_sd for my black box exporter but how to use this for prometheus config yml.can you point to some resource? Thank Eswar -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3e8bb02d-82e3-416d-a862-99e03cb50bde%40googlegroups.com.
[prometheus-users] Too many scrape configs on prometheus config file
Hi, I have a prometheus set up which has too many (roughly 50 static configs) and this number would increase as we have lot of vms still be scraped for prometheus. The file is becoming huge , especially when some applications require basic auth. is there any best practice to maintain the config file as with such huge file some times changes would cause accidental errors. Thanks Eswar -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/65a0249c-c27a-479d-9df0-b68e5102b883%40googlegroups.com.
