To more easily debug your issue, please take Grafana out of the equation, as it has its own foibles. To do this, use the PromQL web browser to formulate a query within the PromQL web interface.
Then, show if there's a difference between the results: if there is, show the exact query you're giving to PromQL and the exact definition of the recording rule. Show both graphs, and highlight the differences. My *guess* is it's something to do with detected counter resets, i.e. example_metrics is not increasing monotonically. You can formulate queries to detect this. On Tuesday, 11 March 2025 at 13:48:54 UTC Kishore Kumar wrote: > Hi Prometheus users, > We are having a PromQL query and a Recording Rule that records > the PromQL, like the example given below. > > *-record: rest-server-recording-rule* > * expr: sum(increase(example_metric[1m])) by (kubernetes_container_name)* > > The Recording Rule Scrape interval and Evaluation interval are both 30 > seconds, and set in Prometheus configuration. > > We are seeing unexpected spikes in the data recorded by the recording > rule, whereas this *unexpected spike* is not present in the source > expression, like shown in the below graph (Used Grafana for comparison). > > Can we know the reason why this unexpected spike is being created by the > recording rule? We would like to know the explanation of how recording rule > captures the data of a query. > > Thanks for reading this message, have a great day. > > *Sum(increase) RawQuery* - data produced when we query the raw > *expression* directly. > *Recording Rules* - Data captured by the recording rule. > [image: image-2025-3-10_18-52-48.png] > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/0ff53a7d-3672-43a3-80aa-008d2b3821b2n%40googlegroups.com.
