yeganeahmadnejad opened a new issue, #13888:
URL: https://github.com/apache/apisix/issues/13888
## Description
We use a priority-based upstream (`nodes[].priority`) with active + passive
health checks to fail over from a primary node to backup nodes when the primary
becomes unhealthy. We're seeing the primary node continue to receive real
traffic for minutes after one nginx worker process has already logged that it
crossed the `unhealthy.http_failures` threshold for that node.
This reproduces consistently on **3.13.0, 3.15.0, and 3.18.0** with an
identical upstream config — it does not appear to be tied to any specific
APISIX version.
## Upstream config (sanitized)
```yaml
upstreams:
- id: upstream-api-service-priority-failover
type: roundrobin
scheme: https
pass_host: node
checks:
active:
type: https
timeout: 2
http_path: /internal-health
https_verify_certificate: false
healthy:
interval: 3
http_statuses: [200, 404]
successes: 2
unhealthy:
interval: 3
http_statuses: [500, 502, 503, 504]
http_failures: 2
tcp_failures: 2
timeouts: 2
passive:
type: https
healthy:
http_statuses: [200, 201]
successes: 2
unhealthy:
http_statuses: [500, 502, 503, 504]
http_failures: 2
tcp_failures: 2
timeouts: 2
nodes:
- host: primary.example.internal
port: 443
weight: 100
priority: 0
- host: fallback-a.example.internal
port: 443
weight: 50
priority: -1
- host: fallback-b.example.internal
port: 443
weight: 50
priority: -1
```
## Reproduction / evidence
Deployment runs with multiple nginx worker processes per pod, and multiple
pod replicas behind a Kubernetes Service. We added `$pid` to our access log
format to correlate real requests with the worker process that handled them,
alongside APISIX's own `[healthcheck]` warn logs (which already include the
worker's OS pid in the `<pid>#<tid>:` prefix).
Example (all identifiers replaced with fakes, but structure/timing is real):
```
2026-08-26T14:37:34Z [warn] 65#65: [lua] healthcheck.lua:1394: log():
[healthcheck]
(upstream#/upstreams/upstream-api-service-priority-failover)
unhealthy HTTP increment (2/2) for '10.0.1.100(10.0.1.100:443)', context:
ngx.timer
```
→ worker process **65** in pod `api-gateway-abc123def-11111` has just
crossed the unhealthy threshold for the primary node.
6.5 minutes later, a real request is served **by the same pod**, but by a
**different worker process (pid 70)**, and is still routed to the same primary
node:
```json
{
"timestamp": "2026-08-26T14:44:04Z",
"pod_name": "api-gateway-abc123def-11111",
"pid": "70",
"upstream_addr": "10.0.1.100:443",
"status": "201"
}
```
No further `[healthcheck]` log lines were emitted for this target on this
pod in between — in particular, no "healthy SUCCESS" recovery line, which
`lua-resty-healthcheck`'s `incr_counter()` would emit on any real state
transition back to healthy (an unhealthy→healthy transition doesn't
short-circuit the way same-state reports do). That suggests worker 65's checker
state genuinely stayed "unhealthy" the whole time, and worker 70 simply never
converged to it.
## Suspected mechanism
`resty.healthcheck`'s `checker:get_target_status()` reads from an
**in-process, per-worker Lua table** (`self.targets`), not shm directly:
```lua
local function get_target(self, ip, port, hostname)
return ((self.targets[ip] or EMPTY)[port] or EMPTY)[hostname]
end
```
`self.targets[...].internal_health` is populated once at checker init (read
from shm) and thereafter updated **only via `worker_events` pub/sub** — each
worker subscribes to health-status-change events raised by whichever worker
currently holds the active-check "leader" lock. If that event is dropped,
delayed, or not processed by a given worker, that worker's local cache never
converges, and it keeps routing traffic to a node that shm — and other workers
— correctly consider unhealthy.
We additionally observed a related but distinct issue on 3.15/3.18 where
`healthcheck_manager.fetch_checker()` can return `nil` on a `resource_ver`
mismatch (composed from `up_conf.resource_version` + `node_ver`), and
`healthcheck_manager.fetch_node_status()` fails open in that case:
```lua
function _M.fetch_node_status(checker, ip, port, hostname)
if not checker or checker.dead then
return true -- treated as healthy
end
...
end
```
We haven't confirmed whether this second path is the actual cause of the
worker-divergence symptom above, or a separate contributing factor — flagging
both since they compound the same user-visible problem (traffic reaching a node
health checks say is down).
## Environment
- APISIX 3.13.0, 3.15.0, and 3.18.0 (reproduces on all three) — same
upstream config on all.
- Multiple nginx worker processes per instance; multiple replicas behind a
k8s Service.
- Priority-based upstream nodes (primary + 2 fallback), active (https) +
passive checks enabled.
## Expected behavior
Once any worker process's active health checker crosses the unhealthy
threshold for a node, all worker processes in that instance should stop routing
traffic to that node until it's confirmed healthy again — health check state
should not diverge silently between workers within the same process group.
## Additional notes
We can provide a minimal `docker-compose`-based reproduction (single
upstream, no plugins, controlled multi-worker `nginx.conf`) if useful — happy
to follow up with that separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
