shlok-srivastava commented on issue #12275:
URL: https://github.com/apache/apisix/issues/12275#issuecomment-4949885590
We hit the same failure mode and captured a CPU profile that confirms the
mechanism, in case it helps others landing here.
**Environment:** APISIX 3.15.0 (official `apache/apisix:3.15.0-ubuntu`),
openresty/1.27.1.2, traditional role, on a 6.12 kernel. Pods peg at their CPU
limit after ~6–14h and start serving 504s.
### Confirming your observation (CPU stays at 100% even with traffic removed)
We saw exactly this. We relabeled a pegged pod out of the Service so it
received zero traffic, and it stayed pegged at the CPU ceiling for 15+ minutes
— the peg is completely traffic-independent, driven by the `ngx.timer`
counter-sync path.
`perf record -F 99 --call-graph dwarf` on one pegged worker (all workers
were hot equally; master/privileged-agent idle):
```
75.6% ngx_shmtx_lock (self)
6.2% ngx_meta_lua_ffi_shdict_get
2.8% ngx_meta_lua_ffi_shdict_store
4.0% ngx_meta_lua_shdict_lookup
4.3% ngx_shmtx_unlock
1.2% ngx_meta_lua_shdict_expire
```
Hot stacks are all `[JIT'd Lua: prometheus counter] →
ngx_meta_lua_ffi_shdict_get/store → ngx_shmtx_lock`. GC is negligible
(libluajit ~0.9%). It happens once the `prometheus-metrics` shared dict is full
(free_size → 0). That matches the flamegraph and strace in the original report
exactly.
### Root cause in our case (3.15+ specific)
The `prometheus-metrics` dict was filling because the built-in LLM
histograms (`llm_latency`, `llm_prompt_tokens`, `llm_completion_tokens`) were
being recorded on **every** request, including non-LLM `traditional_http`
traffic
```lua
-- ngx_tpl.lua: set $llm_time_to_first_token '0';
-- exporter.lua (3.15/3.16):
if llm_time_to_first_token ~= "" then -- '0' ~= "" is always true
metrics.llm_latency:observe(...) -- fires on every request
end
```
This is fixed in **3.17.0** (guard changed to `~= "0"`). (Tracked in #12953
/ #13088.)
### Fixes that should work
- Disable prometheus plugin 🤷
- Upgrade to **3.17.0** (fixes the LLM-metric over-recording at the source).
- Or keep the plugin enabled and bound the cardinality: set
`plugin_attr.prometheus.metrics.<name>.expire` (TTL to evict idle series), trim
`plugin_attr.prometheus.llm_latency_buckets`, and enlarge the
`prometheus-metrics` dict (default is only `10m`).
Haven't tried the last one. I just chose to disable the plugin until we
perform the upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
