Thanks for the great explanation Brian. absent() looks like the right solution but the documentation about empty vectors confuses me slightly. What I would ideally like is to graph the absent() as two distinct values - is there a way to return that as a query or do I have to address this in my graphing tool (Grafana)?
On Tuesday, January 25, 2022 at 7:35:21 PM UTC Brian Candler wrote: > Prometheus does not store or return "null". > > Prometheus will store a "staleness marker" when it performs a scrape and > doesn't find a timeseries (i.e. metric+label set) which was present in the > previous scrape. Internally, it's stored as a special kind of > floating-point NaN. However it's not returned in queries. Instead, the > timeseries just "disappears" as if it didn't exist. > > A timeseries consists of a series of values V1, V2, V3 .. at timestamps > T1, T2, T3 ... So when you perform an instant query for some arbitrary > time T, then prometheus has to look back in time to find the nearest data > point at or before time T. It will look back up to --query.lookback-delta > (by default 5m), but if it finds a staleness marker, it just doesn't return > any value, i.e. that timeseries is excluded from the vector result. This > is all described here > <https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness>. > > If you want to look gaps in a particular timeseries, you can build a query > using absent() > <https://prometheus.io/docs/prometheus/latest/querying/functions/#absent>, > giving a specific timeseries (metric+labels) that you're looking for, and > prometheus will tell you when the time series doesn't exist (i.e. there is > no data point within the previous 5 minutes *or* the series has been marked > stale) > absent{foo{instance="bar",job="baz"}) > > If you want to see when arbitrary timeseries disappear, without hardcoding > specific label sets, you can try something like this: > > foo offset 5m unless foo # it existed 5 minutes ago, but doesn't > exist now > > present_over_time(foo[24h] offset 5m) unless foo # existed any time > in the last 24 hours until 5 minutes ago, but doesn't exist now > > On Tuesday, 25 January 2022 at 18:59:23 UTC [email protected] > wrote: > >> I know in Grafana I can choose to display NULL as zero or as missing but >> I actually want to graph when a value is null - to visualise downtime where >> a series has gaps. Can I craft a query which will spit out two values for >> NULL / not NULL? >> >> Thanks. >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/02221504-3b87-4971-a06e-508016d01082n%40googlegroups.com.
