Hi Brian,
I hope you have a good day. I humbly request to take a look at the
above attached graphs and reply regarding the same.
Apologies and Thank You,
Kishore.
On Wednesday, March 12, 2025 at 7:31:46 PM UTC+5:30 Kishore Kumar wrote:
> Hi Brian,
> We have used Thanos Query UI to query the graph, and we observe
> the same graph that we observed in Grafana. We use the following recording
> rules, albeit with a different name, and without hidden information.
> *-record: rest-server-recording-rule*
> * expr:
> sum(increase(envoy_cluster_upstream_rq{kubernetes_namespace=~".*<hidden>.*",
> kubernetes_pod_name=~"rest-.*", envoy_cluster_name=~"<hidden>"}[3m])/3) by
> (kubernetes_namespace,kubernetes_container_name,envoy_cluster_name)*
>
> The source metric here, envoy_cluster_upstream_rq is not actually
> monotonically increasing graph, and there are counter resets happening.
> Attaching the images below.
> *envoy_cluster_upstream_rq: *
> [image: 2.png]
>
> *Sum of envoy_cluster_upstream_rq: *
> [image: 1.png]
>
> *Actual Query: *
>
> *[image: 3.png]*
>
> *Recording Rule: *
>
> *[image: 4.png]*
>
> Even if it is not a monotonically increasing graph, new spikes should not
> have been created in the recording rule as we don't see them in the actual
> query result.
>
> We would like to know if we are supposed to change any parameters related
> to recording rules, to make them match as close as possible.
>
> Thanks for the response,
> Have a nice day.
> Kishore
>
>
> On Wednesday, March 12, 2025 at 12:33:26 AM UTC+5:30 Brian Candler wrote:
>
>> To more easily debug your issue, please take Grafana out of the equation,
>> as it has its own foibles. To do this, use the PromQL web browser to
>> formulate a query within the PromQL web interface.
>>
>> Then, show if there's a difference between the results: if there is, show
>> the exact query you're giving to PromQL and the exact definition of the
>> recording rule. Show both graphs, and highlight the differences.
>>
>> My *guess* is it's something to do with detected counter resets, i.e.
>> example_metrics is not increasing monotonically. You can formulate queries
>> to detect this.
>>
>> On Tuesday, 11 March 2025 at 13:48:54 UTC Kishore Kumar wrote:
>>
>>> Hi Prometheus users,
>>> We are having a PromQL query and a Recording Rule that records
>>> the PromQL, like the example given below.
>>>
>>> *-record: rest-server-recording-rule*
>>> * expr: sum(increase(example_metric[1m])) by (kubernetes_container_name)*
>>>
>>> The Recording Rule Scrape interval and Evaluation interval are both 30
>>> seconds, and set in Prometheus configuration.
>>>
>>> We are seeing unexpected spikes in the data recorded by the recording
>>> rule, whereas this *unexpected spike* is not present in the source
>>> expression, like shown in the below graph (Used Grafana for comparison).
>>>
>>> Can we know the reason why this unexpected spike is being created by the
>>> recording rule? We would like to know the explanation of how recording rule
>>> captures the data of a query.
>>>
>>> Thanks for reading this message, have a great day.
>>>
>>> *Sum(increase) RawQuery* - data produced when we query the raw
>>> *expression* directly.
>>> *Recording Rules* - Data captured by the recording rule.
>>> [image: image-2025-3-10_18-52-48.png]
>>>
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/f836919b-c18a-48f4-9a3e-49ee005c8391n%40googlegroups.com.
