One thing I forgot to ask:
how bad is that query (since for now it does what is needed)
performance-wise?
my_metric{label="value"}[100y]
On Tue, Jan 5, 2021 at 8:57 PM Roman Dodin <dodin.ro...@gmail.com> wrote:
> Thanks Ben, it is clear now.
> I will check if it makes sense to add a "metric-replay" feature in the
> collector to monotonously export the on_change-type values.
>
>
> On Tue, Jan 5, 2021 at 8:13 PM Ben Kochie <sup...@gmail.com> wrote:
>
>>
>>
>> On Tue, Jan 5, 2021 at 5:36 PM Roman Dodin <dodin.ro...@gmail.com> wrote:
>>
>>> Thank you for your comments, Stuart, maybe I expressed myself a bit
>>> vague.
>>> Let me be more precise and maybe then it will be easier to answer my
>>> question.
>>>
>>> The metric I am talking about is the number of routes a network element
>>> has in its routing table. This integer is reported to the collector only
>>> when it's changed (i.e. a new route has been added to the table and now the
>>> number of routes is X+1).
>>> The collector receives the number of routes and exposes this metric for
>>> Prometheus to scrape. Prometheus successfully scrapes this metric and
>>> stores in TSDB.
>>>
>>> If no more routes are added to the system for a period T, no metrics
>>> will be available for Prometheus to scrape, but at the same time, in my
>>> view, it is not an event, it is a metric, it is just not periodically
>>> reported, because there is no changes to it. We are reducing the amount of
>>> data to transfer and store, by not sending the data that hasn't changed.
>>>
>>> I am curious, if that makes the original question clearer? Is it not
>>> against the design to use Prometheus for such metrics and with this
>>> particular scraping strategy?
>>>
>>
>> Yes, it's more clear. This data behavior is incompatible with Prometheus
>> due to the way stale data is handled. If a thing exists, it should continue
>> to be exported even if it's not changing.
>>
>> Prometheus deals with this just fine, as it uses compression to store the
>> data. In the case of non-changing values, the storage requirements are
>> extremely trivial, on the order of a few bits.
>>
>> In fact, not repeating is almost no savings, as there is a minimum of 120
>> samples stored in a 2 hour window.
>>
>> I suggest changing your exporter behavior to always produce the "current
>> state of the world".
>>
>>
>>>
>>> On Tue, Jan 5, 2021 at 5:14 PM Stuart Clark <stuart.cl...@jahingo.com>
>>> wrote:
>>>
>>>> On 05/01/2021 15:35, Roman Dodin wrote:
>>>>
>>>> Hello community,
>>>> I have a telemetry system that sends data into Prometheus when the data
>>>> changes (i.e. its a push-on-event mechanism, not a sample/interval push)
>>>> That means, if a system is stable, then the last reported value can be
>>>> reported quite some time back.
>>>>
>>>> Let's say this metric is called "my_metric", how do I get its last
>>>> reported value without specifying manually the timeframe to look back?
>>>> So far I come up with the following query:
>>>>
>>>> my_metric{label="value"}[100y]
>>>>
>>>> Is this the only way to get the last value of a series, or maybe there
>>>> is another alternative?
>>>>
>>>> Prometheus is designed to be used as a scrape (pull) system where
>>>> metrics are regularly fetched and their values recorded in the TSDB (even
>>>> if they haven't changed). As a result of this and metric that doesn't have
>>>> a recent value is seen as "stale" and isn't returned when doing instant
>>>> queries. The default setting is 5 minutes, which leads to the generally
>>>> suggested maximum scrape interval of 2 minutes (to allow for a single
>>>> scrape failure without the series being marked as stale).
>>>>
>>>> I'm not sure how you are pushing events into Prometheus, but it sounds
>>>> like you are working against the design of the system. Additionally
>>>> Prometheus is a metrics system rather than an event storage system.
>>>>
>>>> For event storage I would use a different database (possibly SQL based
>>>> or a generic timeseries database) which would have no problems with the
>>>> sort of queries you are wanting.
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/CAFBEvLtEbUhutdYcBc58Zme7tCOT-nVacod2tAWPMuYq954w_Q%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/prometheus-users/CAFBEvLtEbUhutdYcBc58Zme7tCOT-nVacod2tAWPMuYq954w_Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAFBEvLseLU5B%3Dv%2B2KxQ7%3D_2KdyhxhrcPSU-w2PjL9OYtJg6seQ%40mail.gmail.com.
