Re: [prometheus-developers] function last_over_time
On Tue, 28 Apr 2020 at 10:25, Julien Pivotto wrote: > On 28 Apr 10:13, Brian Brazil wrote: > > On Tue, 28 Apr 2020 at 10:06, Julien Pivotto > wrote: > > > > > On 28 Apr 09:51, Brian Brazil wrote: > > > > On Tue, 28 Apr 2020 at 09:33, Julien Pivotto > > > > wrote: > > > > > > > > > Dear all, > > > > > > > > > > We had another request for a promql function, last_over_time. > > > > > > > > > > https://github.com/prometheus/prometheus/issues/7179 > > > > > https://github.com/prometheus/prometheus/pull/6530 > > > > > https://github.com/prometheus/prometheus/pull/1139 > > > > > > > > > > I would like to see more broadly comments about this - it seems > that > > > > > there are usescases for this (I needed this - I was lucky to be > able to > > > > > do it with recording rules - but you can not really do it with > > > > > subqueries). > > > > > > > > > > > > > Thus far we've had no valid concrete use cases. The requests thus far > > > > appear to have been a mix of misunderstanding of PromQL semantics, > and > > > > attempting to use Prometheus as an event logging system. > > > > > > > > What's your use case? > > > > > > > > Brian > > > > > > One of our partners have a flaky API ; we don't always get the data we > > > need, and have gaps of up to 1 hour every now and then. > > > > > > This is gauges (number of credits we still have) so only the last value > > > we could scrape matters. > > > > > > Since it is a third party there isn't much we can do about it. We did > > > not want to hack the exporter neither. We currently have a recording > rule > > > but that has drawbacks, e.g. upon prometheus restart. > > > > > > > For resource limit issues, I'd expect min/max over time to cover that > > sufficiently. I'd also argue that if an hour has passed with no new data > > then either it has already alerted, or it otherwise no longer matters. > > The real issue here is the lack of reliability, and I'm not sure if > trying > > to paper over it really helps. > > > > Brian > > min_over_time and the alerts keeps firing '1 hour' after we add new > resources. > > max_over_time and we are '1 hour' late. > > last_over_time would really be the best compromise we have there. > You could silence the alert, or look at a shorter range for the alert. Predict_linear may also help. I'm reluctant to add features to PromQL to workaround things that only occur when a target is down for an extended period, as the target being down is a bigger problem that should (in principle) be alerting you anyway. Brian > > > > > > > > > > > > > > > > > > > > > > > > > > I know that we are careful when adding new promql functions, but > this > > > > > seems to be a valid one. > > > > > > > > > > Maybe we can revisit the conclusion here and add this promql > function? > > > > > > > > > > > > > > > -- > > > > > (o-Julien Pivotto > > > > > //\Open-Source Consultant > > > > > V_/_ Inuits - https://www.inuits.eu > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Prometheus Developers" group. > > > > > To unsubscribe from this group and stop receiving emails from it, > send > > > an > > > > > email to prometheus-developers+unsubscr...@googlegroups.com. > > > > > To view this discussion on the web visit > > > > > > > > > https://groups.google.com/d/msgid/prometheus-developers/20200428083353.GA36342%40oxygen > > > > > . > > > > > > > > > > > > > > > > > -- > > > > Brian Brazil > > > > www.robustperception.io > > > > > > -- > > > (o-Julien Pivotto > > > //\Open-Source Consultant > > > V_/_ Inuits - https://www.inuits.eu > > > > > > > > > -- > > Brian Brazil > > www.robustperception.io > > -- > (o-Julien Pivotto > //\Open-Source Consultant > V_/_ Inuits - https://www.inuits.eu > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLqVDr%2BdXshFwf110%2BAOE%3D0T4D0a56zMPm9yoJj_guE_sw%40mail.gmail.com.
Re: [prometheus-developers] function last_over_time
On 28 Apr 10:13, Brian Brazil wrote: > On Tue, 28 Apr 2020 at 10:06, Julien Pivotto wrote: > > > On 28 Apr 09:51, Brian Brazil wrote: > > > On Tue, 28 Apr 2020 at 09:33, Julien Pivotto > > wrote: > > > > > > > Dear all, > > > > > > > > We had another request for a promql function, last_over_time. > > > > > > > > https://github.com/prometheus/prometheus/issues/7179 > > > > https://github.com/prometheus/prometheus/pull/6530 > > > > https://github.com/prometheus/prometheus/pull/1139 > > > > > > > > I would like to see more broadly comments about this - it seems that > > > > there are usescases for this (I needed this - I was lucky to be able to > > > > do it with recording rules - but you can not really do it with > > > > subqueries). > > > > > > > > > > Thus far we've had no valid concrete use cases. The requests thus far > > > appear to have been a mix of misunderstanding of PromQL semantics, and > > > attempting to use Prometheus as an event logging system. > > > > > > What's your use case? > > > > > > Brian > > > > One of our partners have a flaky API ; we don't always get the data we > > need, and have gaps of up to 1 hour every now and then. > > > > This is gauges (number of credits we still have) so only the last value > > we could scrape matters. > > > > Since it is a third party there isn't much we can do about it. We did > > not want to hack the exporter neither. We currently have a recording rule > > but that has drawbacks, e.g. upon prometheus restart. > > > > For resource limit issues, I'd expect min/max over time to cover that > sufficiently. I'd also argue that if an hour has passed with no new data > then either it has already alerted, or it otherwise no longer matters. > The real issue here is the lack of reliability, and I'm not sure if trying > to paper over it really helps. > > Brian min_over_time and the alerts keeps firing '1 hour' after we add new resources. max_over_time and we are '1 hour' late. last_over_time would really be the best compromise we have there. > > > > > > > > > > > > > > > > > > I know that we are careful when adding new promql functions, but this > > > > seems to be a valid one. > > > > > > > > Maybe we can revisit the conclusion here and add this promql function? > > > > > > > > > > > > -- > > > > (o-Julien Pivotto > > > > //\Open-Source Consultant > > > > V_/_ Inuits - https://www.inuits.eu > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "Prometheus Developers" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > an > > > > email to prometheus-developers+unsubscr...@googlegroups.com. > > > > To view this discussion on the web visit > > > > > > https://groups.google.com/d/msgid/prometheus-developers/20200428083353.GA36342%40oxygen > > > > . > > > > > > > > > > > > > -- > > > Brian Brazil > > > www.robustperception.io > > > > -- > > (o-Julien Pivotto > > //\Open-Source Consultant > > V_/_ Inuits - https://www.inuits.eu > > > > > -- > Brian Brazil > www.robustperception.io -- (o-Julien Pivotto //\Open-Source Consultant V_/_ Inuits - https://www.inuits.eu -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200428092529.GA106111%40oxygen. signature.asc Description: PGP signature
Re: [prometheus-developers] function last_over_time
On Tue, 28 Apr 2020 at 10:06, Julien Pivotto wrote: > On 28 Apr 09:51, Brian Brazil wrote: > > On Tue, 28 Apr 2020 at 09:33, Julien Pivotto > wrote: > > > > > Dear all, > > > > > > We had another request for a promql function, last_over_time. > > > > > > https://github.com/prometheus/prometheus/issues/7179 > > > https://github.com/prometheus/prometheus/pull/6530 > > > https://github.com/prometheus/prometheus/pull/1139 > > > > > > I would like to see more broadly comments about this - it seems that > > > there are usescases for this (I needed this - I was lucky to be able to > > > do it with recording rules - but you can not really do it with > > > subqueries). > > > > > > > Thus far we've had no valid concrete use cases. The requests thus far > > appear to have been a mix of misunderstanding of PromQL semantics, and > > attempting to use Prometheus as an event logging system. > > > > What's your use case? > > > > Brian > > One of our partners have a flaky API ; we don't always get the data we > need, and have gaps of up to 1 hour every now and then. > > This is gauges (number of credits we still have) so only the last value > we could scrape matters. > > Since it is a third party there isn't much we can do about it. We did > not want to hack the exporter neither. We currently have a recording rule > but that has drawbacks, e.g. upon prometheus restart. > For resource limit issues, I'd expect min/max over time to cover that sufficiently. I'd also argue that if an hour has passed with no new data then either it has already alerted, or it otherwise no longer matters. The real issue here is the lack of reliability, and I'm not sure if trying to paper over it really helps. Brian > > > > > > > > > > > I know that we are careful when adding new promql functions, but this > > > seems to be a valid one. > > > > > > Maybe we can revisit the conclusion here and add this promql function? > > > > > > > > > -- > > > (o-Julien Pivotto > > > //\Open-Source Consultant > > > V_/_ Inuits - https://www.inuits.eu > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Prometheus Developers" group. > > > To unsubscribe from this group and stop receiving emails from it, send > an > > > email to prometheus-developers+unsubscr...@googlegroups.com. > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/prometheus-developers/20200428083353.GA36342%40oxygen > > > . > > > > > > > > > -- > > Brian Brazil > > www.robustperception.io > > -- > (o-Julien Pivotto > //\Open-Source Consultant > V_/_ Inuits - https://www.inuits.eu > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLokGEc%2BBiMPVazOSivx8QBy17zrBM0zsmQkROY3E4gXxA%40mail.gmail.com.
Re: [prometheus-developers] function last_over_time
On 28 Apr 09:51, Brian Brazil wrote: > On Tue, 28 Apr 2020 at 09:33, Julien Pivotto wrote: > > > Dear all, > > > > We had another request for a promql function, last_over_time. > > > > https://github.com/prometheus/prometheus/issues/7179 > > https://github.com/prometheus/prometheus/pull/6530 > > https://github.com/prometheus/prometheus/pull/1139 > > > > I would like to see more broadly comments about this - it seems that > > there are usescases for this (I needed this - I was lucky to be able to > > do it with recording rules - but you can not really do it with > > subqueries). > > > > Thus far we've had no valid concrete use cases. The requests thus far > appear to have been a mix of misunderstanding of PromQL semantics, and > attempting to use Prometheus as an event logging system. > > What's your use case? > > Brian One of our partners have a flaky API ; we don't always get the data we need, and have gaps of up to 1 hour every now and then. This is gauges (number of credits we still have) so only the last value we could scrape matters. Since it is a third party there isn't much we can do about it. We did not want to hack the exporter neither. We currently have a recording rule but that has drawbacks, e.g. upon prometheus restart. > > > > > > I know that we are careful when adding new promql functions, but this > > seems to be a valid one. > > > > Maybe we can revisit the conclusion here and add this promql function? > > > > > > -- > > (o-Julien Pivotto > > //\Open-Source Consultant > > V_/_ Inuits - https://www.inuits.eu > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Prometheus Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to prometheus-developers+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > https://groups.google.com/d/msgid/prometheus-developers/20200428083353.GA36342%40oxygen > > . > > > > > -- > Brian Brazil > www.robustperception.io -- (o-Julien Pivotto //\Open-Source Consultant V_/_ Inuits - https://www.inuits.eu -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200428090633.GA88647%40oxygen. signature.asc Description: PGP signature
Re: [prometheus-developers] function last_over_time
On Tue, 28 Apr 2020 at 09:33, Julien Pivotto wrote: > Dear all, > > We had another request for a promql function, last_over_time. > > https://github.com/prometheus/prometheus/issues/7179 > https://github.com/prometheus/prometheus/pull/6530 > https://github.com/prometheus/prometheus/pull/1139 > > I would like to see more broadly comments about this - it seems that > there are usescases for this (I needed this - I was lucky to be able to > do it with recording rules - but you can not really do it with > subqueries). > Thus far we've had no valid concrete use cases. The requests thus far appear to have been a mix of misunderstanding of PromQL semantics, and attempting to use Prometheus as an event logging system. What's your use case? Brian > > I know that we are careful when adding new promql functions, but this > seems to be a valid one. > > Maybe we can revisit the conclusion here and add this promql function? > > > -- > (o-Julien Pivotto > //\Open-Source Consultant > V_/_ Inuits - https://www.inuits.eu > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/20200428083353.GA36342%40oxygen > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLphb8GPV%2ByK-xGaJBkOq-adOgM60kLvXQvJ8pkK51n%3Ddg%40mail.gmail.com.
