hanzhenfang commented on issue #13385:
URL: https://github.com/apache/apisix/issues/13385#issuecomment-4628006370
Hi @shreemaan-abhishek ,I can reproduce the issue with a minimal Docker
Compose setup on `apache/apisix:3.16.0-debian`.
In this minimal reproduction, the deleted node is still returned by the
health check Control API immediately after it is removed from the upstream
config. It is later purged after the purge window, but during that window the
Control API still exposes the deleted node.
### Environment
- APISIX: `apache/apisix:3.16.0-debian`
- etcd: `bitnamilegacy/etcd:3.6.4`
- Upstream services: `nginx:alpine`
- Health check type: `tcp`
- Deployment mode: traditional / etcd
### docker-compose.yaml
```yaml
services:
etcd:
image: bitnamilegacy/etcd:3.6.4
container_name: apisix-etcd
environment:
- ALLOW_NONE_AUTHENTICATION=yes
- ETCD_ADVERTISE_CLIENT_URLS=http://etcd:2379
ports:
- "127.0.0.1:2379:2379"
apisix:
image: apache/apisix:3.16.0-debian
container_name: apisix
depends_on:
- etcd
- upstream-1
- upstream-2
volumes:
- ./config.yaml:/usr/local/apisix/conf/config.yaml:ro
ports:
- "127.0.0.1:9080:9080"
- "127.0.0.1:9090:9090"
- "127.0.0.1:9180:9180"
upstream-1:
image: nginx:alpine
container_name: upstream-1
ports:
- "127.0.0.1:18083:80"
upstream-2:
image: nginx:alpine
container_name: upstream-2
ports:
- "127.0.0.1:18084:80"
```
### config.yaml
```yaml
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
allow_admin:
- 0.0.0.0/0
admin_key:
- name: admin
key: "my-secret-admin-key"
role: admin
etcd:
host:
- "http://etcd:2379";
apisix:
node_listen: 9080
# For local reproduction only.
# Do not expose Control API publicly in production.
enable_control: true
control:
ip: "0.0.0.0"
port: 9090
```
### Steps to reproduce
Start from a clean environment:
```bash
docker compose down --remove-orphans
docker compose up -d
```
Create an upstream with two nodes and active TCP health check:
```bash
curl -i http://127.0.0.1:9180/apisix/admin/upstreams/issue-13385-demo \
-H "X-API-KEY: my-secret-admin-key" \
-H "Content-Type: application/json" \
-X PUT \
-d '{
"type": "roundrobin",
"nodes": {
"upstream-1:80": 1,
"upstream-2:80": 1
},
"checks": {
"active": {
"type": "tcp",
"timeout": 1,
"healthy": {
"interval": 1,
"successes": 2
},
"unhealthy": {
"interval": 1,
"tcp_failures": 2,
"timeouts": 3
}
}
}
}'
```
Create a route using this upstream:
```bash
curl -i http://127.0.0.1:9180/apisix/admin/routes/issue-13385-demo \
-H "X-API-KEY: my-secret-admin-key" \
-H "Content-Type: application/json" \
-X PUT \
-d '{
"uri": "/issue-13385-demo",
"upstream_id": "issue-13385-demo",
"plugins": {
"proxy-rewrite": {
"uri": "/"
}
}
}'
```
Trigger the upstream once so the health checker is initialized:
```bash
curl -i http://127.0.0.1:9080/issue-13385-demo
sleep 3
```
Check the health check state:
```bash
curl -sS http://127.0.0.1:9090/v1/healthcheck/upstreams/issue-13385-demo
```
At this point, both nodes are returned.
Now remove `upstream-1:80` from the upstream config:
```bash
curl -i http://127.0.0.1:9180/apisix/admin/upstreams/issue-13385-demo \
-H "X-API-KEY: my-secret-admin-key" \
-H "Content-Type: application/json" \
-X PUT \
-d '{
"type": "roundrobin",
"nodes": {
"upstream-2:80": 1
},
"checks": {
"active": {
"type": "tcp",
"timeout": 1,
"healthy": {
"interval": 1,
"successes": 2
},
"unhealthy": {
"interval": 1,
"tcp_failures": 2,
"timeouts": 3
}
}
}
}'
```
Confirm the Admin API only contains one node:
```bash
curl -sS http://127.0.0.1:9180/apisix/admin/upstreams/issue-13385-demo \
-H "X-API-KEY: my-secret-admin-key"
```
The upstream config now contains only:
```json
"nodes": {
"upstream-2:80": 1
}
```
Immediately query the health check state again:
```bash
curl -sS http://127.0.0.1:9090/v1/healthcheck/upstreams/issue-13385-demo
```
### Actual behavior
The deleted node is still returned by the Control API health check endpoint.
In my local run:
```text
upstream-1 -> 172.22.0.2
upstream-2 -> 172.22.0.4
```
After deleting `upstream-1:80`, the Admin API only showed `upstream-2:80`,
but the health check endpoint still returned both nodes:
```json
{
"nodes": [
{
"port": 80,
"ip": "172.22.0.2",
"hostname": "172.22.0.2",
"status": "healthy",
"purge_time": 1780631859.964
},
{
"port": 80,
"ip": "172.22.0.4",
"hostname": "172.22.0.4",
"status": "healthy",
"purge_time": 1780631859.964
}
],
"type": "tcp",
"name": "/apisix/upstreams/issue-13385-demo"
}
```
After hitting the route again, the current node no longer had `purge_time`,
but the deleted node was still visible:
```json
{
"nodes": [
{
"port": 80,
"ip": "172.22.0.4",
"hostname": "172.22.0.4",
"status": "healthy"
},
{
"port": 80,
"ip": "172.22.0.2",
"hostname": "172.22.0.2",
"status": "healthy",
"purge_time": 1780631928.128
}
],
"type": "tcp",
"name": "/apisix/upstreams/issue-13385-demo"
}
```
After waiting longer, the deleted node was eventually purged in this minimal
setup:
```bash
sleep 15
curl -sS http://127.0.0.1:9090/v1/healthcheck/upstreams/issue-13385-demo
```
Result:
```json
{
"nodes": [
{
"port": 80,
"ip": "172.22.0.4",
"hostname": "172.22.0.4",
"status": "healthy"
}
],
"type": "tcp",
"name": "/apisix/upstreams/issue-13385-demo"
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
