On 10/03/2020 07:47, Venkata Bhagavatula wrote:
Hi Stuart, Julien,

Following is being done in the application side for the metrics given in the above mails.:

Some of the metrics used in these charts have multiple labels. Due to the usage of multiple labels and the possible different values of these labels, the cardinality of the metrics can be very high. So to avoid an exponential growth of number of metrics combination that Prometheus ends up scrapping, the application cleans up counters that are not incremented for a some period . So at some point some of the metrics (which have some current value) are removed.

can this be the reason why we see a drop of the counter value in the above charts ?



Counters must not decrease unless due to a service restart or overflow event. Otherwise the counter logic will assume the decrease is actually due to a reset/overflow and rate() will return a spike at that point.


Thanks n Regards,

Chalapathi.








On Mon, Mar 9, 2020 at 5:09 PM Julien Pivotto <roidelapl...@inuits.eu <mailto:roidelapl...@inuits.eu>> wrote:

    On 09 Mar 11:35, Stuart Clark wrote:
    > On 2020-03-09 11:21, Venkata Bhagavatula wrote:
    > > Hi Stuart,
    > >
    > > sorry for the late reply, i was on vacation.
    > > I will check the recording rule.
    > > The reduction was same for both recorded metric vs original
    metric.
    > > can you correct my understanding?
    > > After the rule is evaluated, will the type of metric be treated as
    > > Gauge?
    > >
    >

    Hi there,

    Please note that metric types in Prometheus are informative. You
    can in
    theory run rate() on both gauges and counters, even if you should only
    do it on counters.

    Regards

--  (o-    Julien Pivotto
     //\    Open-Source Consultant
     V_/_   Inuits - https://www.inuits.eu

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com <mailto:prometheus-users+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABXnQPs2HbKE7puPY3dMkABmODrve56UYWhDPW7GZuHWfs8gww%40mail.gmail.com <https://groups.google.com/d/msgid/prometheus-users/CABXnQPs2HbKE7puPY3dMkABmODrve56UYWhDPW7GZuHWfs8gww%40mail.gmail.com?utm_medium=email&utm_source=footer>.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/44609b83-d350-a382-38e6-6ee8a72ee0d3%40Jahingo.com.

Reply via email to