aydemirkala commented on issue #13961:
URL: https://github.com/apache/apisix/issues/13961#issuecomment-6186033838
**Confirming this on 3.15 → 3.18, with two control groups**
We hit the same symptom upgrading from 3.15 to 3.18 (20 replicas,
`limits.cpu: 4`, `worker_processes: auto`). The apisix container saturates
its full 4-core quota and hits **100% CFS throttling** in short bursts,
roughly hourly but **not wall-clock aligned** — consistent with
`ngx.timer.every` counting from worker start rather than a fixed clock.
Two controls in the same deployment isolate the cause:
1. A single pod still on **3.15** (`nginx-lua-prometheus-api7 0.20240201`,
which has no `remove_expired_keys` / `flush_expired`) — not affected.
2. A pod on **3.18 with the prometheus plugin removed** — not affected.
Client-side impact: upstream calls that APISIX logs with
`$request_time=0.040` are observed as **5+ seconds** by the calling
service's APM. `ListenOverflows` / `ListenDrops` are **0** on every pod,
so the accept queue is not overflowing — the workers are stalled. That
matches lock contention in the `log` phase, which happens after
`$request_time` has stopped counting.
One more detail: `remove_expired_keys_interval` cannot be tuned from
APISIX. `exporter.lua` calls
`base_prometheus.init("prometheus-metrics", metric_prefix)` with a string
prefix, so the library takes the `else` branch and pins the interval to
`MAX_REMOVE_EXPIRED_KEYS_INTERVAL` (3600). Passing an options table would
at least let operators spread the reclaim out.
Is there a plan to revisit #13981? It addresses exactly the unbounded-lock
property we are seeing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
