On 14/06/2022 18:32, Jeremy Collette wrote:
Hello,
We have written a custom exporter that exposes metrics with explicit
timestamps, which Prometheus periodically scrapes. In the case where
Prometheus becomes temporarily unavailable, these metric samples will
be cached in the exporter until they are scraped, causing affected
metrics to age.
I understand that if a metric is older than a certain threshold, it
will be rejected by Prometheus with the message: "Error on ingesting
samples that are too old or are too far into the future".
I'm trying to understand if there are any guarantees surrounding the
ingestion of historical metrics. Is there some metric sample age that
is guaranteed to be recent enough to be ingested? For example, are
samples with timestamps within the last hour always going to be
considered recent? Within the last five minutes?
According to this previous thread: Error on ingesting samples that are
too old
<https://groups.google.com/g/prometheus-users/c/rKJYm6naEow/m/zylud_J4AAAJ>,
MR seems to indicate that metrics as old as 1 second can be dropped
due to being too old. Is this interpretation correct? If so, is there
any way to ensure metrics with timestamps won't be dropped for being
too old?
The use of timestamps in metrics is not something that should be used
except in some very specific cases. The main use case for adding a
timestamp is when you are scraping metrics into Prometheus that have
been sourced from another existing metrics system (for example things
like the Cloudwatch Exporter). You also mention something about your
exporter caching things until they are scraped, which also sounds like
something that is not advisable. The action of the exporter shouldn't
really be changing depending on the requests being received (or not
received).
An exporter is expected to return the various metrics that reflect
"now", in the same way that a directly instrumented application would be
expected to return the current state of the metrics being maintained in
memory. For a simple exporter the normal mechanism is for a request to
be received which then triggers some mechanism to generate the metrics.
For example with something like the MySQL Exporter a request would
trigger a query on the connected database which then returns various
information that is converted into Prometheus metrics and returned. In
some situations the process to fetch information from the underlying
system can be quite resource intensive or slow. In that case a common
design is to decouple the information fetching process from the request
handling process. One example is to perform the information fetching
process on a periodic timer, with the information fetched then stored in
memory. The request process then reads and returns that information -
returning the same values for every request until the next cycle of the
information fetching process. In none of these standard scenarios would
you expect timestamps to be attached to the returned metrics.
It would be good to hear a bit more about what you are trying to do, as
it is highly likely that the use of timestamps in your use case is
probably not the right option and they should just be dropped.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/007cf462-d87e-d4c7-316e-4007567c74a1%40Jahingo.com.
