sawyerward opened a new issue, #14050:
URL: https://github.com/apache/apisix/issues/14050
### Current Behavior
One of three APISIX replicas began returning 503 for one service after its
Endpoints watch failed during a Kubernetes API server restart. It kept doing so
for about 3.5 hours, until we restarted the pod. The service's pods were
healthy the whole time, and the other two replicas served it normally. At least
one other kubernetes-discovery service on the same replica kept working.
We haven't confirmed the cause, but this code in
`apisix/discovery/kubernetes/informer_factory.lua` looks related. `list()`
fetches later pages with a recursive call and ignores its return value:
```lua
informer.continue = data.metadata.continue
if informer.continue and informer.continue ~= "" then
if informer.stop then
return true
end
list(httpc, apiserver, informer)
end
return true
```
If a later page fails, `list()` would still return `true`, and `post_list()`
would then delete the endpoints from the pages that never loaded. The retry was
due about 40 seconds after the watch failure, at about 21:40:57 UTC. The API
servers were returning etcd errors from 21:39:48 to 21:41:18, and etcd had no
leader from 21:40:59 to 21:41:05, so a later page could have failed. Our
cluster has about 1,170 Endpoints, which is about 10 pages of 120. In today's
list order, the lost service is on a later page than a service that kept
working. We don't know the order at the time of the incident, or whether other
services were also removed.
### Expected Behavior
A failed re-list should not remove endpoints, or should be retried until it
completes.
### Error Logs
The only discovery error on the affected replica between 21:39 and 21:50 UTC
(we collect error-level logs only):
```
[error] informer_factory.lua:335: watch failed, kind: Endpoints, reason:
ReadBodyError, message : closed, context: ngx.timer
```
From 21:41:04 until the restart, every request to the service failed with:
```
handle_upstream(): failed to set upstream: no valid upstream node: nil
```
No `list failed` or `connect apiserver failed` error was logged.
### Steps to Reproduce
We hit this once in production and haven't reproduced it. What led up to it:
1. Run APISIX with `discovery.kubernetes` (single cluster, Endpoints) in a
cluster with more than 120 Endpoints, so the list spans several pages.
2. Route to a service through kubernetes discovery.
3. Restart the Kubernetes API servers so the Endpoints watch fails and
APISIX re-lists while the API servers are returning errors. For us this was a
managed AKS control-plane restart, during which etcd briefly lost its leader.
4. One replica returned 503 with `no valid upstream node` for one service
until the pod was restarted.
### Environment
- APISIX version (run `apisix version`): 3.18.0
(`apache/apisix:3.18.0-ubuntu`, Helm chart 2.17.0). The code above is the same
in 3.19.0 and master.
- Operating system (run `uname -a`): Linux 6.8.0-1064-azure #72-Ubuntu SMP
Mon Jul 20 21:42:25 UTC 2026 x86_64 GNU/Linux (Ubuntu 24.04.4)
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
openresty/1.29.2.4
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`): not used (`config_provider: yaml`,
standalone mode). Kubernetes is AKS v1.35.4.
- APISIX Dashboard version, if relevant: N/A
- Plugin runner version, for issues related to plugin runners: N/A
- LuaRocks version, for installation issues (run `luarocks --version`): N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
