Baoyuantop commented on issue #12449: URL: https://github.com/apache/apisix/issues/12449#issuecomment-4600122143
Thanks for the additional information @wangchao732 . I checked the current issue details and the APISIX Prometheus plugin code path. `apisix_http_status` is a counter in APISIX, and the plugin increments it once in the HTTP log phase for the matched route/service/status. The screenshots show that the Grafana `sum(rate(...))` query has an abnormal spike, but they are not enough to determine whether APISIX is over-counting, Prometheus/Grafana is producing a scrape/rate artifact, or some internal calls are still going through APISIX. Could you please provide the following details? 1. The full route/service/upstream configuration for the service or route that shows the spike, not only the global rule. 2. The number of APISIX pods/instances, the Prometheus scrape interval, the actual `$__rate_interval` value in Grafana, and whether APISIX pods restarted or scaled during the spike window. 3. Raw `apisix_http_status` samples around the spike, preferably grouped by `instance`, `route`, `service`, and `code`, instead of only the aggregated `sum(rate(...))` graph. 4. The APISIX access log count for the same time window and how that count was calculated. 5. Whether the Dapr/internal service calls from the upstream application also pass through APISIX. With those details, we can continue verifying this on APISIX 3.11 and determine whether this is an APISIX counting issue or a monitoring aggregation/scrape issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
