wangchao732 commented on issue #12449:
URL: https://github.com/apache/apisix/issues/12449#issuecomment-4601225171
@Baoyuantop Very glad to receive your reply. Best wishes! Details：
Service info:
{
"name": "dapr-service-vehicle-realtime-data",
"upstream": {
"nodes": [
{
"host": "10.233.31.69",
"port": 80,
"weight": 1
}
],
"timeout": {
"connect": 60,
"send": 60,
"read": 60
},
"type": "roundrobin",
"hash_on": "vars",
"scheme": "http",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
}
}
Routers info:
1、
{
"uris": [
"/dapr-service-vehicle-realtime-data/*",
"/realtime/*"
],
"name": "dapr-service-vehicle-realtime-data-https",
"host": "xxx",
"plugins": {
"ext-plugin-pre-req": {
"allow_degradation": false,
"conf": [
{
"name": "TokenValidator",
"value":
"{\"validate_header\":\"token\",\"rejected_code\":\"403\"}"
}
],
"disable": false
}
},
"service_id": "420094949904614258",
"status": 1
}
2、
{
"uris": [
"/dapr-service-vehicle-realtime-data/info/query",
"/info/query",
"/realtime/getLastestInfos",
"/dapr-service-vehicle-realtime-data/realtime/getLastestInfos",
"/realtime/getLastLocation",
"/dapr-service-vehicle-realtime-data/realtime/getLastLocation",
"/dapr-service-vehicle-realtime-data/info/updatePositioningSysMode",
"/info/updatePositioningSysMode",
"/info/updateParam",
"/info/delParam",
"/dapr-service-vehicle-realtime-data/info/updateParam",
"/dapr-service-vehicle-realtime-data/info/delParam"
],
"name": "dapr-service-vehicle-realtime-data-info-query",
"methods": [
"POST"
],
"host": "xxx",
"service_id": "420094949904614258",
"status": 1
}
<img width="2227" height="699" alt="Image"
src="https://github.com/user-attachments/assets/76656c0b-c312-43bd-9781-5dfb62c5035f";
/>
Promethues config :
- job_name: "apisix"
scrape_interval: 30s
metrics_path: "/apisix/prometheus/metrics"
static_configs:
- targets: ["xxx:30215"] # apisix promethues expose port number.
Deployment:
spec:
replicas: 2
selector:
matchLabels:
app: dapr-service-vehicle-realtime-data
template:
metadata:
creationTimestamp: null
labels:
app: dapr-service-vehicle-realtime-data
annotations:
dapr.io/app-id: dapr-service-vehicle-realtime-data
dapr.io/app-port: '8080'
dapr.io/config: zipkin
dapr.io/enabled: 'true'
Grafana $__rate_interval: last 6 hours . When I manually set the Grafana
query range to 5 minutes and also change the query expression to a fixed value
(e.g., “5 minutes”), the issue persists; it recovers automatically after about
10–15 seconds, and a spike occurs approximately every 1–2 hours.
sum(rate(apisix_http_status{service=~"$service",route=~"$route",instance=~"$instance"}[5m]))
by (service)
<img width="2552" height="627" alt="Image"
src="https://github.com/user-attachments/assets/804f9074-5ef8-4365-bc95-dc880079507b";
/>
<img width="1914" height="1227" alt="Image"
src="https://github.com/user-attachments/assets/48d8957f-57fa-402d-bbbb-2861f21f8b51";
/>
Alert config:
- name: apisix_total_requests
rules:
- alert: apisix_total_requests
expr: sum(rate(apisix_http_status [5m])) > 100000
for: 3m
labels:
severity: "P1"
annotations:
summary: "Apisix requests above 100000."
description: "Attention! Host={{ $labels.instance }},name={{
$labels.name }},env={{ $labels.env }},service={{ $labels.service }},process={{
$labels.groupname }} Apisix request above 1000 for 5 minutes (current value: {{
$value }})"
APISIX access log count ： sed -n '/2026-06-02 17:38:40/,/2026-06-02
17::39:39/p' apisix*.log The total number is far less than the peak.
Dapr/internal service calls ： internal
dns,例如：dapr-service-vehicle-realtime-data.dapr-application.svc.cluster.local:xxx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
