Prometheus doesn't support storing historical data and samples with
out-of-order timestamps. If you need writing such data, then take a look at
other Prometheus-inspired solutions. See, for example,
https://victoriametrics.github.io/#backfilling .
On Mon, Jul 6, 2020 at 10:31 AM Venkata Bhagavatula
wrote:
> Hi All,
> Can any one respond to my queries? Also we observed the following:
> 1. If for eg timestamp(epoch) in the scrape is 12:00:00, then prometheus
> is not scrapping the targets
> 2. If for eg timestamp(epoch) in the scrape is 12:00:01, then prometheus
> is scrapping the targets.
>
> Thanks & regards,
> Chalapathi
>
> On Thu, Jul 2, 2020 at 3:03 PM Venkata Bhagavatula <
> venkat.cha...@gmail.com> wrote:
>
>> Hi,
>>
>> We are using prometheus version 2.11.1, In our application, the scrape
>> target has timestamp field. when timestamp field is present, then
>> prometheus is not scrapping any metrics.
>> Following is the output of the curl request for scrape target:
>>
>>- *cmd: curl http://:24231/metrics*
>>
>> meas_gauge{id="Filtered",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="Rejected",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="ReprocessedIn",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="Created",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="Duplicated",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>> meas_gauge{id="Stored",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 336.0 159368040
>> meas_gauge{id="Retrieved",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 354.0 159368040
>> meas_gauge{id="ReducedInMerging",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
>> 0.0 159368040
>>
>>
>>
>>- I checked that time is in sync between the prometheus node and the
>>target node.
>>- Following is the epoch time on the prometheus node:
>>
>> *cmd: date +'%s%3N'*
>> *1593681793979*
>>
>>
>>- Epoch difference between the prometheus node and the time stamp
>>present in the sample is more than an hour.
>>
>> difference = ( 1593681793979 - 159368040) / 1000 = 1393sec = 23min
>>
>> Scrape_interval is configured as 300s
>> honor_timestamps is set to true.
>>
>> Can you let us know why prometheus is not able to scrape the targets? Is
>> it due to the timestamp difference between prometheus and target?
>> How much difference will prometheus tolerate?
>>
>> Thanks n Regards,
>> Chalapathi
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CABXnQPuB5iWDhDw06OLOepmz5_XgC2a%3DC9uuVaDKcczR9B-%2BAA%40mail.gmail.com
>
> .
>
--
Best Regards,
Aliaksandr Valialkin, CTO VictoriaMetrics
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAPbKnmCdo2txis5J8vVRGbMGB4X1N%2BjQiUr9aujrf1sqdXmjPw%40mail.gmail.com.
