[prometheus-users] prometheus not scrapping targets when timestamp field is present

[Venkata Bhagavatula]

Hi,

We are using prometheus version 2.11.1, In our application, the scrape
target has timestamp field.  when timestamp field is present, then
prometheus is not scrapping any metrics.
Following is the output of the curl request for scrape target:

   - *cmd: curl  http://<readacted>:24231/metrics*

meas_gauge{id="Filtered",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0
1593680400000
meas_gauge{id="Rejected",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0
1593680400000
meas_gauge{id="ReprocessedIn",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
0.0 1593680400000
meas_gauge{id="Created",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 0.0
1593680400000
meas_gauge{id="Duplicated",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
0.0 1593680400000
meas_gauge{id="Stored",HOST="test",STREAM="Smoke_stream",NODE="MFE2"} 336.0
1593680400000
meas_gauge{id="Retrieved",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
354.0 1593680400000
meas_gauge{id="ReducedInMerging",HOST="test",STREAM="Smoke_stream",NODE="MFE2"}
0.0 1593680400000



   - I checked that time is in sync between the prometheus node and the
   target node.
   - Following is the epoch time on the prometheus node:

*cmd: date +'%s%3N'*
*1593681793979*


   - Epoch difference between the prometheus node and the time stamp
   present in the sample is more than an hour.

difference = ( 1593681793979 -  1593680400000) / 1000 = 1393sec = 23min

Scrape_interval is configured as 300s
honor_timestamps is set to true.

Can you let us know why prometheus is not able to scrape the targets? Is it
due to the timestamp difference between prometheus and target?
How much difference will prometheus tolerate?

Thanks n Regards,
Chalapathi

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABXnQPtZiz%3DTvhYjGHuonVParMt68pKSa-qdUS6D_2GB--xPgA%40mail.gmail.com.