AlinsRan opened a new issue, #14103:
URL: https://github.com/apache/apisix/issues/14103
### Current Behavior
`publish_stream_bytes` keeps, for each `(listen_addr, service, service_id)`
slot of `apisix_stream_metrics_zone`, a baseline of how much of the zone total
has been added to `apisix_stream_bandwidth`. The baselines live in the
`prometheus-metrics` shared dict, next to the counters. Since #14101, a slot
without a baseline is counted from zero (`published = dict:get(key) or 0`).
When `prometheus-metrics` is full, shdict writes can forcibly evict other
valid entries (LRU). If a baseline is evicted while its counter survives, the
next read takes the slot as new and adds its whole lifetime total to the
counter again, even without new traffic. Each further loss of the baseline
replays it once more.
Two things make this more likely than it looks:
- Every collection touches the baselines first
(`collect_stream_zone_metrics`) and the counters after them
(`prometheus:metric_data()` reads every key). So the baselines sit behind the
counters in the LRU queue and are evicted first.
- Writing the replayed baseline back needs memory in the same full dict,
which can evict another baseline from the tail. While the dict stays full, this
may keep replaying on every collection. This last part is inferred from the
code and not yet confirmed on a running gateway.
Before #14101, a missing baseline was retaken without counting, so the same
eviction only lost one collection interval. #14101 turns it into an overcount
of the slot's full total, documented there as a known trade-off.
### Expected Behavior
Losing a baseline should not replay the slot's total. Bytes already counted
should not be counted again, and at worst one interval may be lost, as before.
### Error Logs
When the dict is full, `nginx-lua-prometheus` logs LRU eviction / `no
memory` errors for the `prometheus-metrics` dict. Nothing is logged about the
replay itself; it shows up as a spike in `apisix_stream_bandwidth`.
### Steps to Reproduce
1. Set `nginx_config.meta.lua_shared_dict.prometheus-metrics` to a small
size (e.g. `1m`), enable the stream `prometheus` plugin and
`apisix_stream_metrics_zone`, and add a stream route bound to a service.
2. Send one 5-byte session and scrape the metrics endpoint: the counter of
the svc slot is 5.
3. Fill the dict (e.g. `dict:set("filler" .. i, string.rep("x", 200))` until
`forcible` is returned), and check with `dict:get_keys(0)` (not `get`, which
would touch the LRU) that the baseline key is gone while the counter key is
still there.
4. Scrape again without sending traffic: the counter is expected to become
10 while the zone total is still 5.
5. Repeat 3–4: 15, 20, ...
A focused LuaJIT check of `publish_stream_bytes` with dict/counter
substitutes shows the counter going 15 → 30 → 45 with the zone total fixed at
15 and the baseline removed twice (see
https://github.com/apache/apisix/pull/14101#issuecomment-6436506934). The steps
above have not been run against a real gateway yet.
Possible fixes:
- Keep the baselines in a dedicated shared dict and write them with
`safe_set`, so they are never evicted by other metrics. If that dict is full,
skip the slot and log, which undercounts instead of replaying.
- Or keep state that tells a new slot apart from a lost baseline.
The regression test should cover repeated baseline loss while the counter
remains and the zone total does not change.
### Environment
- APISIX version: master with #14101 applied
- Affects the stream `prometheus` plugin with `apisix_stream_metrics_zone`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
