Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
On Friday, 12 August 2022 at 08:50:28 UTC+1 ninag...@gmail.com wrote: > I'm worried about the blackbox_exporter is expensive or not. > > You are, or are not, worried about this? > I also have other jobs, other jobs are with global 30s scrape intervals > and 30s evaluation invervals, when I set the special job with scrape > interval 2m, will it cause any problem? > > Each job defines its own scrape interval. The global default is used only if it doesn't define its own scrape interval. Similarly, each alerting rule group defines its own evaluation interval. The global default is used only if it doesn't define its own evaluation interval. Scraping and rule evaluation run entirely independently. > " > If you only want to *check* a value every 1 day for alerting, then it's > perfectly fine to have an alerting rule group with a 1 day evaluation > interval - even if the underlying metric is collected more frequently. The > alerting rule will use the most recent value at evaluation time. > " > regarding this point, is it possible to set alerting rule group for a > special rule or a job? > I don't understand the question. You can create as many alerting rule groups as you like, and each rule group can have as many alerting rules as you like. Each alerting expression can access *all* the metrics in the TSDB - it is not tied to a specific scrape job. https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/848984ad-044c-47db-a19a-326c6a536528n%40googlegroups.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
Thank you. Brian. I'm worried about the blackbox_exporter is expensive or not. I also have other jobs, other jobs are with global 30s scrape intervals and 30s evaluation invervals, when I set the special job with scrape interval 2m, will it cause any problem? " If you only want to *check* a value every 1 day for alerting, then it's perfectly fine to have an alerting rule group with a 1 day evaluation interval - even if the underlying metric is collected more frequently. The alerting rule will use the most recent value at evaluation time. " regarding this point, is it possible to set alerting rule group for a special rule or a job? On Friday, August 12, 2022 at 3:40:23 PM UTC+8 Brian Candler wrote: > Perhaps the most import question to ask is: *why* do you only want to > scrape once per day? > > If you're worrying about storage space consumption: don't. Prometheus > uses differential encoding. If you record the same metric repeatedly with > the same value, i.e. the difference between the adjacent values is zero, > then the amount of storage space used is miniscule. Even if it's not the > same value, it's extremely highly compressed. And disk space is cheap. > > If you only want to *check* a value every 1 day for alerting, then it's > perfectly fine to have an alerting rule group with a 1 day evaluation > interval - even if the underlying metric is collected more frequently. The > alerting rule will use the most recent value at evaluation time. > > If you're worried that the exporter itself is expensive to run on each > scrape, then you should do one of the other things that Stuart suggested > (i.e. using textfile collector or push gateway). On the target node use a > cron job to update the metric at whatever interval makes sense - but let > prometheus scrape it as often as it likes. But you should still scrape the > cached value every 1-2 minutes. > > If it's blackbox_exporter that you're talking about, those scrapes are > cheap. So you may as well scrape every 2 minutes. > > If you're worried about transient errors causing spurious alarms, then > adjust your alerting rules (e.g. in the simplest case, add "for: 30m" which > means that you'll only get a notification if the alert has been active for > 30m *continuously*). > > On Friday, 12 August 2022 at 08:31:09 UTC+1 Stuart Clark wrote: > >> On 12/08/2022 08:29, nina guo wrote: >> > So if with push gateway or textfile collector, we need to also to >> > customize our metrics, am I right? >> >> What do you mean? >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0a952d82-c1d7-4ad1-9978-20b84985b698n%40googlegroups.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
Perhaps the most import question to ask is: *why* do you only want to scrape once per day? If you're worrying about storage space consumption: don't. Prometheus uses differential encoding. If you record the same metric repeatedly with the same value, i.e. the difference between the adjacent values is zero, then the amount of storage space used is miniscule. Even if it's not the same value, it's extremely highly compressed. And disk space is cheap. If you only want to *check* a value every 1 day for alerting, then it's perfectly fine to have an alerting rule group with a 1 day evaluation interval - even if the underlying metric is collected more frequently. The alerting rule will use the most recent value at evaluation time. If you're worried that the exporter itself is expensive to run on each scrape, then you should do one of the other things that Stuart suggested (i.e. using textfile collector or push gateway). On the target node use a cron job to update the metric at whatever interval makes sense - but let prometheus scrape it as often as it likes. But you should still scrape the cached value every 1-2 minutes. If it's blackbox_exporter that you're talking about, those scrapes are cheap. So you may as well scrape every 2 minutes. If you're worried about transient errors causing spurious alarms, then adjust your alerting rules (e.g. in the simplest case, add "for: 30m" which means that you'll only get a notification if the alert has been active for 30m *continuously*). On Friday, 12 August 2022 at 08:31:09 UTC+1 Stuart Clark wrote: > On 12/08/2022 08:29, nina guo wrote: > > So if with push gateway or textfile collector, we need to also to > > customize our metrics, am I right? > > What do you mean? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/635f5d4d-ff92-45b0-b7ef-291930df5b3cn%40googlegroups.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
now I'm using blackbox exporter exposed metrics. if i use pushgateway or textfile collector, i need to write a script first to generate the metrics and write them with prometheus metrics format to textfile collector folder. On Friday, August 12, 2022 at 3:31:09 PM UTC+8 Stuart Clark wrote: > On 12/08/2022 08:29, nina guo wrote: > > So if with push gateway or textfile collector, we need to also to > > customize our metrics, am I right? > > What do you mean? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d4fc5e45-a324-47f0-855e-4e482cdfbac0n%40googlegroups.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
now I'm using blackbox exporter exposed metrics. if i use pushgateway or textfile collector, i need to write a script first to generate the metrics and write them with prometheus metrics format to textfile collector folder. I cannot use the metrics exposed by blackbox exporter. On Friday, August 12, 2022 at 3:31:09 PM UTC+8 Stuart Clark wrote: > On 12/08/2022 08:29, nina guo wrote: > > So if with push gateway or textfile collector, we need to also to > > customize our metrics, am I right? > > What do you mean? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/866c2386-10fc-4e15-9632-716d9865391bn%40googlegroups.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
On 12/08/2022 08:29, nina guo wrote: So if with push gateway or textfile collector, we need to also to customize our metrics, am I right? What do you mean? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/999dfca4-1bdc-a1dc-6135-12a5d532a0fe%40Jahingo.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
So if with push gateway or textfile collector, we need to also to customize our metrics, am I right? On Friday, August 12, 2022 at 3:17:09 PM UTC+8 Stuart Clark wrote: > On 12/08/2022 08:09, nina guo wrote: > > OK. So if I want to scrape the metrics for 1 day interval, which way > > is better to implement? > > Some options would be: > > - Scrape it every 2 minutes instead of daily > - Use the textfile collector of the node exporter, with a scheduled job > to update the file daily > - Use the push gateway with a scheduled job that updates the API daily > > For the second two options you will lose the ability to use the "up" > metric (as that will now refer to the node exporter/push gateway > instead), but both add their own additional metrics containing timstamps > of the last time the metric was updated. > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ca5f7551-2dd3-46de-9331-c12e2f12cd29n%40googlegroups.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
On 12/08/2022 08:09, nina guo wrote: OK. So if I want to scrape the metrics for 1 day interval, which way is better to implement? Some options would be: - Scrape it every 2 minutes instead of daily - Use the textfile collector of the node exporter, with a scheduled job to update the file daily - Use the push gateway with a scheduled job that updates the API daily For the second two options you will lose the ability to use the "up" metric (as that will now refer to the node exporter/push gateway instead), but both add their own additional metrics containing timstamps of the last time the metric was updated. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ca768714-190e-93da-84ff-66ce4878d2d4%40Jahingo.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
OK. So if I want to scrape the metrics for 1 day interval, which way is better to implement? On Friday, August 12, 2022 at 2:56:29 PM UTC+8 Stuart Clark wrote: > On 12/08/2022 06:46, nina guo wrote: > > what i want to implement is to scrape and evaluate one kind of metrics not > so frequently, I want to adjust the interval to 1d or 2d, something like > this. > > On Friday, August 12, 2022 at 11:06:15 AM UTC+8 nina guo wrote: > >> Hi, I received following error. >> >> - job_name: TLS Connection >> scrape_interval: 1d >> evaluation_interval: 1d >> metrics_path: /probe >> params: >> module: [smtp_starttls] >> file_sd_configs: >> - files:xxx >> >> kubectl logs prometheus -c prometheus -n monitoring >> level=error ts=2022-08-12T03:03:50.120Z caller=main.go:347 msg="Error >> loading config (--config.file=/etc/prometheus/prometheus.yml)" err="parsing >> YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line >> 54: field evaluation_interval not found in type config.ScrapeConfig" >> > Two things here: > > 1. There is no entry called "evaluation_interval" within a scrape config, > so that needs removing to clear the unmarshal error. > > 2. The maximum sensible scrape interval is around 2-3 minutes, so 1 day is > far too long. With a longer interval you will end up with stale time series > and "gaps" in all your graphs. > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/592becb6-6ed5-4320-aa0e-be25c783c7ecn%40googlegroups.com.
Re: [prometheus-users] Re: evaluation interval cannot be recognized in seperate job
On 12/08/2022 06:46, nina guo wrote: what i want to implement is to scrape and evaluate one kind of metrics not so frequently, I want to adjust the interval to 1d or 2d, something like this. On Friday, August 12, 2022 at 11:06:15 AM UTC+8 nina guo wrote: Hi, I received following error. - job_name: TLS Connection scrape_interval: 1d evaluation_interval: 1d metrics_path: /probe params: module: [smtp_starttls] file_sd_configs: - files:xxx kubectl logs prometheus -c prometheus -n monitoring level=error ts=2022-08-12T03:03:50.120Z caller=main.go:347 msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" err="parsing YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line 54: field evaluation_interval not found in type config.ScrapeConfig" Two things here: 1. There is no entry called "evaluation_interval" within a scrape config, so that needs removing to clear the unmarshal error. 2. The maximum sensible scrape interval is around 2-3 minutes, so 1 day is far too long. With a longer interval you will end up with stale time series and "gaps" in all your graphs. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/903f4c47-1a79-fc56-6f71-039caf0aa400%40Jahingo.com.
[prometheus-users] Re: evaluation interval cannot be recognized in seperate job
what i want to implement is to scrape and evaluate one kind of metrics not so frequently, I want to adjust the interval to 1d or 2d, something like this. On Friday, August 12, 2022 at 11:06:15 AM UTC+8 nina guo wrote: > Hi, I received following error. > > - job_name: TLS Connection > scrape_interval: 1d > evaluation_interval: 1d > metrics_path: /probe > params: > module: [smtp_starttls] > file_sd_configs: > - files:xxx > > kubectl logs prometheus -c prometheus -n monitoring > level=error ts=2022-08-12T03:03:50.120Z caller=main.go:347 msg="Error > loading config (--config.file=/etc/prometheus/prometheus.yml)" err="parsing > YAML file /etc/prometheus/prometheus.yml: yaml: unmarshal errors:\n line > 54: field evaluation_interval not found in type config.ScrapeConfig" > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/045485e8-e2a3-438d-af8d-83079fd4cca3n%40googlegroups.com.
