moonming opened a new pull request, #14167: URL: https://github.com/apache/apisix/pull/14167
### Description A worker of the kubernetes discovery can get stuck serving stale pod IPs for hours (#14149). Two things combine: 1. `update_endpoint_dict` wrote the `<svc>#version` key **before** the `<svc>` content key. A worker that read the version between the two writes picked up the new version with the old content. The two `safe_set` calls are each atomic, but nothing orders them for a concurrent reader. 2. `core.lrucache` (without `invalid_stale`) hands an expired entry back as long as its version still matches, so that worker never re-read the content until the service changed again. For a quiet service that is never. This PR removes the version key instead of reordering the writes, so there is no write order left to get wrong and nothing in a worker trusts a version any more: - The shared dict holds **one key per service**, written by a single `safe_set`. The dict is the only source of truth. - A worker's lrucache entry lives for **1 second** and is then re-read from the dict (`invalid_stale = true`). A miss (unknown service) is cached for the same time so it does not hit the dict on every request. - The multi-cluster path (`discovery_args.cluster_ids`) merges the selected dicts on the same schedule instead of concatenating per-cluster versions. - The tars discovery used the same two-key pattern and gets the same change. - The `#version` filter in `dump_endpoints_from_dict` is kept so that keys left in the shared dict by an older build after a hot reload stay invisible; the first relist removes them as dirty keys. Trade-off: a change to an Endpoints object reaches a worker within one second instead of on the next request. The cost of the re-read is one `cjson` decode per active service per worker per second, measured at about 18 µs for a 72-node service (2.7 µs for 10 nodes, 224 µs for 1000 nodes). The consul discovery already uses this model (`ttl = 1`, `invalid_stale = true`, no version key) since #13066. Behaviour that is unchanged: the discovery readiness check, the `dump` control API output, the `discovery_args.cluster_ids` semantics, and the relist / dirty-key cleanup. #### Tests - `t/kubernetes/discovery/kubernetes6.t` runs against the real apiserver of the kubernetes CI job: it changes the `ns-a/ep` Endpoints through the kubectl proxy and sends real requests through APISIX to the fake backends on 1980/1981, so every worker's cache is exercised. It covers publishing a service, switching backends within the ttl with no answer from outside the two backends during the switch, scaling to zero and back, deleting and recreating the Endpoints object, an unknown service, and that the dump contains no version keys. - `t/kubernetes/discovery/kubernetes7.t` runs without an apiserver: it writes the shared dict directly to show that an entry is re-read once it expires, and asserts that the writer publishes exactly one key per service. - The tests have not been run locally (no working OpenResty on this machine); they rely on the CI run. #### Which issue(s) this PR fixes: Fixes #14149 ### Checklist - [x] I have explained the need for this PR and the problem it solves - [x] I have explained the changes or the new features added to this PR - [x] I have added tests corresponding to this change - [ ] I have updated the documentation to reflect this change (no configuration, API or dump format changed) - [x] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
