Removing the metric data shouldn't be done.
The expectation is that you can query an exporter at any point and receive
metrics. Ideally that would be instant data from the time of the scrape, but
for processes that might only happen occasionally or queries which take a while
to produce you should return the latest result.
Prometheus needs to get data regularly (generally no more infrequently as every
2 minutes as previously mentioned). Scrapes which return the same value as the
previous one are compressed heavily, so there is no real worries about storage
space.
If a metric stops being scraped it becomes stale, and stops being returned as
you have seen.
As a second point regarding exporters never removing state after a state part
of the design for Prometheus is the way HA works is achieved by having both
servers of a HA pair scrape each target. If exporters removed any state after a
scrape this would break. Also it is very useful to be able to query an exporter
manually if debugging potential metrics issues.
So in summary the recommendation would be to increase the scrape period to no
slower than 2 minutes and adjust your exporter to always return the latest set
of metrics, with that state being updated every 15 minutes as you do currently.
On 23 January 2021 09:29:20 GMT, akshay sharma <sharmaakshay...@gmail.com>
wrote:
>But my exporter is exporting the data every 15min after some processing
>,and we are deleting the data as soon as Prometheus scrapes to avoid
>registering of same metric after 15minutes,as this a limitation with
>Prometheus go client. We need to unregister to avoid registering of
>same
>metric.
>
>If Prometheus scrape every 1minutes, then next minute there will be no
>data
>till next 15minute.
>
>How can we resolve this sync issue?
>
>On Sat, Jan 23, 2021, 2:49 PM Ben Kochie <sup...@gmail.com> wrote:
>
>> You need to scrape data faster than 5 minutes. 2 minutes is the
>> recommended max to avoid stale data. Prometheus gets less efficient
>when
>> you scrape slower than 1 minute.
>>
>> On Sat, Jan 23, 2021 at 8:14 AM akshay sharma
><sharmaakshay...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I've one some queries wrt to prometheus scraping and evaluating
>intervals.
>>>
>>> *Queries as follows.*
>>> 1)
>>> Say a prometheus job with *scrape interval 10 mins and evaluation
>>> interval 5 mins*, has scraped metrics at at t1, when the same is
>*queried
>>> on prometheus UI, as an instant vector, it is visible for around 3
>minutes
>>> 30 seconds* (Note:*Here I am querying repeatedly for 3 minutes 30
>>> seconds*), at say approximately 4th minute, when I query the same *I
>am
>>> seeing no data. And even if I query the same for a range vector
>>> metric_name[5m]*, I don't see the metrics (Note: Here, prometheus
>says,
>>> last s crape was around 4 minutes ago), but the I see the metrics
>when I
>>> give metric_name[10m]. Please explain the behaviour.
>>>
>>> No data, when executing *metric_name(instant vector)*
>>> ElementValue
>>> *no data*
>>>
>>> 2)
>>>
>>> This is* my Prometheus alert rule*
>>>
>>> throughput{instance="x.x.x.x:xxxx"} > 100
>>>
>>> After *alert** fired once, when the metric data missing for more
>than 1
>>> minute, then alert will resolved.* Is there any way to avoid this?
>>> Basically, *I do not want a resolved alert, when there is no
>metrics.*
>>>
>>>
>>> 3)
>>> Say, *there are 2 jobs, for one job, I need almost continuous
>>> monitoring, so, scrape interval is 1 min, and global evaluation
>interval is
>>> 1 min, there is another job which scrapes metrics at an interval of
>10
>>> mins, and since evaluation interval is always global, I think
>evaluation
>>> for the second job also is happening every 1 min, even though there
>are no
>>> metrics scraped, and the evaluation results in no_data *?. This will
>>> cause a problem in alerting as I mentioned above. *Isn't it possible
>to
>>> have evaluation interval specific to job? Request you to provide
>some
>>> solution*.
>>>
>>>
>>> *Thanks,*
>>>
>>> *Akshay*
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>Groups
>>> "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>send an
>>> email to prometheus-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>>
>https://groups.google.com/d/msgid/prometheus-users/CAOrgXNKcJOUib%3D8XXS-TZe3%3DSiczMQK7-8vs4x%3DQ1AnQ9dcakQ%40mail.gmail.com
>>>
><https://groups.google.com/d/msgid/prometheus-users/CAOrgXNKcJOUib%3D8XXS-TZe3%3DSiczMQK7-8vs4x%3DQ1AnQ9dcakQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>
>--
>You received this message because you are subscribed to the Google
>Groups "Prometheus Users" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to prometheus-users+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/prometheus-users/CAOrgXNJKT%2BBz%2Bi%2B9S2uWgX7t1WErDTO1K1U0iPNFJMf0uUgdUg%40mail.gmail.com.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7B6476D1-946A-4E51-A3B4-322695E86BA0%40Jahingo.com.
