pubyun opened a new issue, #3411:
URL: https://github.com/apache/apisix-dashboard/issues/3411
## Summary
APISIX Dashboard 3.0.x can leak etcd logical watchers when store watches
fail and are reinitialized — an application-level watch lifecycle leak. A
**read-only** dashboard can exhaust etcd memory/CPU with no high-rate writes
and a tiny dataset.
## Environment
- `apache/apisix-dashboard` 3.0.0 (`manager-api` 3.0.0); checked `v3.0.1`
has the same relevant code.
- Embedded `go.etcd.io/etcd/client/v3` v3.5.5, connecting directly to etcd.
## Root cause (`api/internal/core/store/store.go`, `api/cmd/root.go`)
1. A failed watch loop appends its `GenericStore` to a global
`storeNeedReInit` slice — **no dedup, never cleared, no mutex**.
2. `ReInit()` runs ~every 2 minutes, iterates the slice, and calls
`store.Init()`.
3. `GenericStore.Init()` → `listAndWatch()` overwrites `s.cancel` **without
cancelling the previous watch context**.
4. The health checker also force-calls `Init()` per store on transient etcd
issues.
With repeated `required revision has been compacted` / transient errors,
live logical watchers accumulate in the Go client. Because `clientv3`
multiplexes logical watchers over a few gRPC Watch streams, etcd shows a
distinctive signature: low `watch_stream_total`, ~flat
`grpc_server_started_total{Watch}`, but rapidly rising
`grpc_server_msg_received_total{Watch}` and `etcd_debugging_mvcc_watcher_total`.
## Production evidence
- Dashboard RSS grew to ~46 GiB; no restart for 186 days.
- The connected etcd member reached **millions** of watchers, ~25 CPU cores
in GC, and OOM/restart loops.
- Dashboard logs: repeated `etcd watch error: key: /apisix/... err:
etcdserver: mvcc: required revision has been compacted` followed by `etcd watch
exception closed, restarting: resource: ...`.
- **Scaling the dashboard to 0 immediately stopped the WatchCreate storm;
etcd `watcher_total` dropped back to single digits.**
## Proposed fix
Make reinitialization idempotent and cancel superseded watches:
- Replace the global slice with a mutex-protected **set**; **drain** it in
`ReInit()`; **requeue only failed** entries.
- Assign a **generation** to each watch; only enqueue reinit when the
current generation exits unexpectedly (`generation == current && !closing &&
ctx.Err() == nil`). This avoids misclassifying deliberate cancels and avoids
re-enqueue churn.
- `Close()` must remove the store from the pending set and mark it closed;
`Init()`/`ReInit()` must check closed so a closed store is not resurrected.
- Cancel the superseded watch (either cancel-old-then-relist, or
new-then-cancel-old). If you keep new-before-cancel, guard event apply by
revision so late events from the old watch can't overwrite cache during the
dual-watch window.
- Add **per-store backoff** for permanently-failing reinit to avoid a tight
2-minute retry + log spam.
## Impact
Production stability: a read-only dashboard can exhaust the shared etcd's
memory/CPU. Upgrading 3.0.0 → 3.0.1 is **not** a fix (same code). We mitigated
by scaling the dashboard to zero.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
