moonming commented on issue #2708: URL: https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-6731837474
@vortegatorres Thanks for the additional details. This **may share the same root cause** as https://github.com/apache/apisix/issues/14149. In the affected APISIX code, Kubernetes discovery writes the shared-dict version key before the node list. A worker reading between those writes can cache the old nodes under the new version. With an unchanged version, the 300-second LRU entry is renewed after expiry instead of re-reading the node list. That worker can therefore keep using deleted Pod IPs until another endpoint change, a worker restart, or cache eviction. This could explain why only some APISIX workers/pods remain stale while others work normally, even though controller configuration pushes are accepted. It is a plausible match for your latest evidence, but we have **not confirmed that your incidents have the same cause**. The fix is merged in https://github.com/apache/apisix/pull/14167 (commit `22ecbc60e4ea780abc0c76ee6396583a26ff7e68`). It removes the separate version key and makes each worker re-read the node list after a one-second cache TTL (`invalid_stale = true`). Could you please validate with an **APISIX gateway build containing that patch**, or a release that includes it, and repeat the backend rollout/scale-down scenario under traffic? The patch needs to be present in the gateway itself. The `3.19.0` release tag still contains the old cache implementation. If the problem persists, please capture the relevant `endpoints` entries from `GET /v1/discovery/kubernetes/dump` on both an affected and an unaffected APISIX pod, together with the corresponding current Kubernetes Endpoints. Please share only the relevant endpoint entries, with sensitive addresses redacted, and omit the dump's `config` section. This would help distinguish stale shared discovery data from a worker continuing to use an old cached node list. We will keep this issue open pending your validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
