Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-05-12 Thread via GitHub 


vortegatorres commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-4435006610

   Following up on your earlier suggestion (@AlinsRan):
   
   > Using health check ups can alleviate this issue.
   
   We enabled active health checks on the affected upstream about 3 weeks ago. 
The same underlying behavior described in this issue persists, and the health 
check has introduced an additional failure mode.
   
   ## Environment
   
   APISIX Helm chart 2.13.0, standalone mode, backends listening on port 8080.
   
   ## Example 
   
   Example of the situation that keeps happening with` IP-A`:
   - APISIX access logs show `IP-A:8080` serving successful responses (status 
200) up to a specific minute.
   - Shortly after, the IP stopped appearing in any backend workload's logs.
   - The Pod was terminated as part of the rolling deploy.
   - APISIX continued active health-checking `IP-A:8080` for approximately 85 
minutes after the Pod no longer existed.
   
   Any update about this issue?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-04-16 Thread via GitHub 


vortegatorres commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-4263259344

   Many thanks for sharing your findings, @jotasixto. In our case, that does 
not appear to be the issue, since we are using port 8080 in all of the cases 
mentioned.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-04-09 Thread via GitHub 


jotasixto commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-4218338000

   We experienced a very similar issue in our environment and wanted to share 
our findings, as the root cause in our case turned out to be related to **EKS 
security group rules blocking low-port traffic between worker nodes**, rather 
than a bug in APISIX or the ingress controller itself.
   
   ## Our Environment
   
   - **EKS cluster** deployed via the official [terraform-aws-eks 
module](https://github.com/terraform-aws-modules/terraform-aws-eks)
   - **APISIX standalone** deployed with Helm chart **v2.13.0**
   - Backend services exposing **port 80** on their pods (front-end Nginx 
containers with `containerPort: 80`)
   
   ## Observed Behavior
   
   We saw the same symptoms described in this issue: after pod rescheduling or 
scaling events, APISIX gateways would intermittently fail to reach backend pods 
with `(111: Connection refused)` errors, particularly when pods were placed on 
different nodes than the APISIX gateway pods.
   
   When we attempted a workaround of ensuring at least one replica of each 
backend service ran on every node, we discovered the actual underlying problem: 
**traffic on port 80 was being blocked between worker nodes by the EKS node 
security group rules**.
   
   ## Root Cause: EKS Security Group Default Rules and Low Ports
   
   The [official terraform-aws-eks documentation on network 
connectivity](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/network_connectivity.md)
 explains that the default node security group rules only allow traffic on 
**ephemeral ports (1025-65535)** between the cluster control plane and worker 
nodes, and between nodes themselves. This is by design — AWS considers it a 
best practice because **non-privileged pods should not bind to ports below 
1024**.
   
   Looking at the [security group 
diagram](https://raw.githubusercontent.com/terraform-aws-modules/terraform-aws-eks/master/.github/images/security_groups.svg)
 from the module documentation, port 80 is simply not in the allowed range for 
node-to-node or cluster-to-node ingress traffic.
   
   This means:
   - When an APISIX gateway pod on **Node A** tries to reach a backend pod on 
**Node B** using port 80, the traffic is **silently dropped** by the node 
security group.
   - When both APISIX and the backend pod happen to be on the **same node**, 
traffic works fine (it stays within the node and doesn't cross the security 
group boundary).
   - This creates the **intermittent** behavior: it works or fails depending on 
pod placement, which changes with scaling events, node rotation, etc.
   
   ## Our Fix
   
   We added custom security group rules to explicitly allow port 80 traffic 
between worker nodes 
([Automya/claims#185](https://github.com/Automya/claims/pull/185)):
   
   ```yaml
   node_security_group_additional_rules:
 ingress_node_ports_fronts:
   description   : "Allow port 80 from cluster to worker 
nodes"
   protocol  : "tcp"
   from_port : 80
   to_port   : 80
   type  : "ingress"
   source_cluster_security_group : true
 ingress_node_ports_fronts_self:
   description   : "Allow port 80 between worker nodes"
   protocol  : "tcp"
   from_port : 80
   to_port   : 80
   type  : "ingress"
   self  : true
   ```
   
   After applying these rules, the issue was **fully resolved** — APISIX 
gateways could reach backend pods on any node in the cluster without 
`Connection refused` errors, regardless of pod placement.
   
   ## Recommendation
   
   If you're running on EKS (especially with the terraform-aws-eks module) and 
your backend services expose pods on **port 80 or any port below 1024**, check 
your node security group rules. The default rules only allow ephemeral ports 
(1025-65535), and traffic to low ports between nodes will be silently dropped.
   
   The proper long-term fix is to **migrate backend services to listen on high 
ports (≥1024)**, which aligns with AWS best practices for non-privileged pods. 
The custom security group rules above are a valid workaround if migrating ports 
immediately is not feasible.
   
   **TL;DR**: In our case, APISIX was working correctly — it was the EKS node 
security groups blocking cross-node traffic on port 80 that caused the 
intermittent `Connection refused` errors after pod rescheduling.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
u

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-11 Thread via GitHub 


AlinsRan commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3888337417

   
   # Issue Investigation Checklist
   
   ## 1. Logs and Runtime Information
   
   ### 1.1 Complete Logs
   
   Please provide the full logs during the time when the issue occurred.
   
    Ingress Pod
   - ingress container logs
   - adc-server container logs
   - If possible, please enable debug logging and reproduce the issue.
   
   Debug configuration references:
   - Ingress debug:  
 
https://github.com/apache/apisix-helm-chart/blob/9d5ad2a28f7e75490e05f9dd977cd29791f13629/charts/apisix-ingress-controller/values.yaml#L80
   - ADC debug:  
 
https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/values.yaml#L77
   
    APISIX Pod
   - `error.log`
   - `access.log` entries that include `/apisix/admin/configs` API calls
   
   ---
   
   ## 2. Restart Verification
   
   Please confirm:
   
   - Does restarting `ingress-apisix` resolve the issue?
   
   This helps determine whether the problem originates from the ingress/ADC 
side or the APISIX data plane.
   
   ---
   
   ## 3. Standalone API Configuration Comparison
   
   When the issue occurs:
   
   - Call the standalone API
   - Retrieve the data plane configuration **before and after restart**
   - Compare the configuration differences
   
   Standalone API reference:  
   https://apisix.apache.org/zh/docs/apisix/deployment-modes/#example
   
   This step is critical for root cause analysis.
   
   ---
   
   ## 4. Resource Limits and Usage
   
   Please provide:
   
   - Resource requests/limits for ingress pods
   - Resource requests/limits for APISIX pods
   - CPU and memory usage when the issue occurs
   
   Insufficient resources may cause sync failures or abnormal behavior.
   
   ---
   
   # 2. Configuration Conflict Investigation
   
   ## 2.1 GatewayProxy Conflict
   
   Please check whether multiple `GatewayProxy` resources are pointing to the 
same data plane.
   
   You can run:
   
   ```bash
   kubectl get gatewayproxy -A -o yaml
   
   
   Multiple `GatewayProxy` resources sharing the same data plane may cause 
configuration conflicts.
   
   ---
   
   ## 2.2 Multiple Ingress Controllers
   
   Please confirm:
   
   * Whether multiple `apisix-ingress-controller` instances are deployed in the 
same Kubernetes cluster
   * If yes, whether they are properly isolated (e.g., ingressClass, namespace 
scope, etc.)
   
   Improper isolation may also lead to configuration conflicts.
   
   ---
   
   # 3. Investigation Goals
   
   The goal is to determine:
   
   * Whether the issue is on the ingress/ADC side or the APISIX data plane side
   * Whether there is a configuration conflict
   * Whether improper resource limits are causing synchronization failures
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-11 Thread via GitHub 


vortegatorres commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3887638182

   > I would like to confirm again whether restarting APISIX alone can resolve 
this issue, and if restarting ingress-apisix can achieve the same effect.
   
   Restarting APISIX definitely works; I didn't try restarting the 
`ingress-controller` because I didn't have the chance. Something new to add 
here is that I have observed that it can auto-recover if I don't restart APISIX 
for some time, sometimes hours.
   
   > When the issue occurs, could you obtain the relevant data of the service 
before and after the restart via the standalone API and provide it for 
subsequent analysis?
   
   There is nothing relevant in the `ingress-controller` logs when it suddenly 
auto-recovered, just this message that is logged every minute:
   ```
   2026-02-11 13:28:41.961 info {
 "level": "info",
 "message": "PUT /sync",
 "requestId": "594c8636-c0d2-43d3-ba37-689ce866ea4e",
 "timestamp": "2026-02-11T12:28:41.961Z"
   } 
   
   2026-02-11 13:28:41.955 info 2026-02-11T12:28:41.955ZINFO
provider.client client/client.go:201syncing resources for config
{"service_number": 89} 
   
   2026-02-11 13:28:41.954 info 2026-02-11T12:28:41.953ZINFO
provider.client client/client.go:177syncing all resources 
   
   2026-02-11 13:27:41.965 info {
 "level": "info",
 "message": "PUT /sync",
 "requestId": "e803b466-d422-469d-9444-ac347be15d61",
 "timestamp": "2026-02-11T12:27:41.965Z"
   } 
   
   2026-02-11 13:27:41.956 info 2026-02-11T12:27:41.956ZINFO
provider.client client/client.go:201syncing resources for config
{"service_number": 89} 
   
   2026-02-11 13:27:41.954 info 2026-02-11T12:27:41.953ZINFO
provider.client client/client.go:177syncing all resources 
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-11 Thread via GitHub 


AlinsRan commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3882943161

   > It recovers after restarting the apisix deployment.
   
   
   I would like to confirm again whether restarting APISIX alone can resolve 
this issue, and if restarting ingress-apisix can achieve the same effect.
   
   When the issue occurs, could you obtain the relevant data of the service 
before and after the restart via the standalone API and provide it for 
subsequent analysis?
   
   Reference document for the standalone API: 
https://apisix.apache.org/zh/docs/apisix/deployment-modes/#example
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-05 Thread via GitHub 


vortegatorres commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3852697800

   > Using health check ups can alleviate this issue.
   
   I will try this then.
   
   > I think you can solve this problem first and then continue to observe.
   
   I applied the fix defined in that issue, and the number of errors dropped 
drastically, but still it see few of these errors:
   ```
   2026-02-05 11:29:27.240 error 2026-02-05T10:29:27.240Z   ERROR   
providerapisix/provider.go:282  failed to sync  {"error": "failed to 
sync 1 configs: GatewayProxy//apisix-config"}
   
   2026-02-05 11:29:27.238 error 2026-02-05T10:29:27.238Z   ERROR   
provider.client client/client.go:210failed to sync resources
{"name": "GatewayProxy//apisix-config", "error": "ADC execution 
errors: [ADC execution error for GatewayProxy//apisix-config: 
[ServerAddr: 
http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,
 Err: connect ECONNREFUSED :9180]]"}
   
   2026-02-05 11:29:27.238 error 2026-02-05T10:29:27.238Z   ERROR   
provider.client client/client.go:269failed to execute adc command   
{"config": 
{"name":"GatewayProxy//apisix-config","serverAddrs":["http://:9180","http://:9180","http://:9180","http://:9180","http://:9180"],"tlsVerify":false},
 "error": "ADC execution error for GatewayProxy//apisix-config: 
[ServerAddr: 
http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,
 Err: connect ECONNREFUSED :9180]"}
   
   2026-02-05 11:29:27.238 error 2026-02-05T10:29:27.238Z   ERROR   
provider.executor   client/executor.go:142  failed to run http sync for 
server  {"server": 
"http://:9180,http://:9180,http://:9180,http://:9180,http://:9180",
 "error": "ServerAddr: 
http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,
 Err: connect ECONNREFUSED :9180"}
   
   2026-02-05 11:29:27.238 error 2026-02-05T10:29:27.237Z   ERROR   
provider.executor   client/executor.go:328  ADC Server sync failed  
{"result": 
{"status":"partial_failure","total_resources":5,"success_count":4,"failed_count":1,"success":[{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"0001-01-01T00:00:00Z","synced_at":"2026-02-05T10:29:23Z","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"0001-01-01T00:00:00Z","synced_at":"2026-02-05T10:29:23Z","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"0001-01-01T00:00:00Z","synced_at":"2026-02-05T10:29:23Z","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"0001-01-01T00:00:00Z","synced_at":"2026-02-05T10:29:23Z","response":{"status":0,"headers":null}}],"failed":[{"event":{"resourceTyp
 
e":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-05T10:29:27.237Z","synced_at":"0001-01-01T00:00:00Z","reason":"connect
 ECONNREFUSED :9180","response":{"status":0,"headers":null}}]}, 
"error": "ADC Server sync failed: connect ECONNREFUSED :9180"}
   ```
   
   > How many APISIX Pods are running?
   
   Between 4 and 15.
   
   > Are there any CPU or memory limits configured for APISIX or 
apisix-ingress-controller?
   > Have you observed the CPU and memory usage of both APISIX and 
apisix-ingress-controller around the time when the issue occurs?
   
   Yes, we have limits, but during the incidents, the CPU and memory usages 
were very low.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-04 Thread via GitHub 


AlinsRan commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3848565194

   > How can I reduce the interval?
   
   Endpoint updates will also immediately trigger synchronization.
   
   
https://github.com/apache/apisix-helm-chart/blob/7e5b8dff478dcc86373d6bd9071ab5444c97602b/charts/apisix-ingress-controller/values.yaml#L96
   
   > INFO   provider.client client/client.go:177syncing all resources 
   
   It indicates that the synchronization is in progress, not that it has been 
completed successfully.
   If the synchronization succeeds, APISIX should not retain IPs of Pods that 
have already been deleted.
   
   >  use of Upstream health-checks 
   
   Using health check ups can alleviate this issue.
   
   > socket hang up
   
   I think you can solve this problem first and then continue to observe.
   
   > It recovers after restarting the apisix deployment.
   
   What I find confusing is why restarting APISIX resolves the issue, whereas 
restarting apisix-ingress-controller does not. This seems to suggest that the 
problem may not be solely on the controller side.
   
   To better understand the situation, could you please share more details 
about your deployment?
   * How many APISIX Pods are running?
   * Are there any CPU or memory limits configured for APISIX or 
apisix-ingress-controller?
   * Have you observed the CPU and memory usage of both APISIX and 
apisix-ingress-controller around the time when the issue occurs?
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-03 Thread via GitHub 


vortegatorres commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3844077083

   > When the issue occurs, are there any error logs in the Ingress Controller?
   
   Sorry, I missed this earlier. This isn't happening just during the incident, 
 `ingress-controller` is consistently logging this error:
   ```
   2026-02-03 23:26:15.434 error 2026-02-03T22:26:15.434Z   ERROR   
providerapisix/provider.go:282  failed to sync  {"error": "failed to 
sync 1 configs: GatewayProxy/k6-api-prod/apisix-config"} 
   
   2026-02-03 23:26:15.432 error 2026-02-03T22:26:15.431Z   ERROR   
provider.client client/client.go:210failed to sync resources
{"name": "GatewayProxy/k6-api-prod/apisix-config", "error": "ADC execution 
errors: [ADC execution error for GatewayProxy/k6-api-prod/apisix-config: 
[ServerAddr: 
http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,
 Err: socket hang up]]"} 
   
   2026-02-03 23:26:15.431 error 2026-02-03T22:26:15.431Z   ERROR   
provider.client client/client.go:269failed to execute adc command   
{"config": 
{"name":"GatewayProxy/k6-api-prod/apisix-config","serverAddrs":["http://:9180","http://:9180","http://:9180","http://:9180","http://:9180","http://:9180"],"tlsVerify":false},
 "error": "ADC execution error for GatewayProxy/k6-api-prod/apisix-config: 
[ServerAddr: 
http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,
 Err: socket hang up]"} 
   
   2026-02-03 23:26:15.431 error 2026-02-03T22:26:15.431Z   ERROR   
provider.executor   client/executor.go:142  failed to run http sync for 
server  {"server": 
"http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,http://:9180",
 "error": "ServerAddr: 
http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,http://:9180,
 Err: socket hang up"} 
   
   2026-02-03 23:26:15.431 error 2026-02-03T22:26:15.431Z   ERROR   
provider.executor   client/executor.go:328  ADC Server sync failed  
{"result": 
{"status":"all_failed","total_resources":6,"success_count":0,"failed_count":6,"success":[],"failed":[{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket
 hang 
up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket
 hang 
up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket
 hang 
up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced
 _at":"0001-01-01T00:00:00Z","reason":"socket hang 
up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket
 hang 
up","response":{"status":0,"headers":null}},{"event":{"resourceType":"","type":"","resourceId":"","resourceName":""},"failed_at":"2026-02-03T22:26:15.431Z","synced_at":"0001-01-01T00:00:00Z","reason":"socket
 hang up","response":{"status":0,"headers":null}}]}, "error": "ADC Server sync 
failed: socket hang up"} 
   ```
   
   > If endpoint updates are delayed, this situation may occur temporarily. 
Could you clarify whether the problem **recovers automatically after some 
time**, or if it **persists for an extended period**?
   
   It recovers after restarting the apisix deployment.
   
   > If it persists for a long time, does **reducing the interval of scheduled 
full synchronization** help mitigate the issue, or does the problem remain even 
after multiple full sync cycles?
   
   How can I reduce the interval?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-02 Thread via GitHub 


AlinsRan commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3836461970

   When the issue occurs, are there any error logs in the Ingress Controller?
   
   If endpoint updates are delayed, this situation may occur temporarily.  
   Could you clarify whether the problem **recovers automatically after some 
time**, or if it **persists for an extended period**?
   
   If it persists for a long time, does **reducing the interval of scheduled 
full synchronization** help mitigate the issue, or does the problem remain even 
after multiple full sync cycles?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-02 Thread via GitHub 


vortegatorres commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3833975822

   As mentioned earlier, the issue is intermittent and not deterministic. In 
almost all cases where we observed the replica count scaling up and down, 
everything worked as expected. Because of that, I can’t provide additional 
reproduction steps beyond what I already did and what you’ve already tried.
   
   Regarding the logs: during the incident, APISIX first reported a `(111: 
Connection refused)` while connecting to the upstream:
   
   ```text
   2026-01-29 04:05:10.638 error 2026/01/29 03:05:10 [error] 49#49: *1392456 
connect() failed (111: Connection refused) while connecting to upstream, 
client: XX, server: _, request: "POST /api HTTP/1.1", upstream: 
"http://YY:8000/api";, host: "url"
   ```
   
   Right after that, the request was retried and succeeded. The key detail is 
that `upstream_addr` contains two IPs: it first tried a stale IP, then retried 
using the IP that worked:
   
   ```json
   2026-01-29 04:05:11.430 {
 "ts": "2026-01-29T03:05:11+00:00",
 "service": "apisix",
 "resp_body_size": "0",
 "host": "url",
 "address": "X",
 "request_length": "595",
 "method": "POST",
 "uri": "/api",
 "status": "204",
 "user_agent": "Go-http-client/2.0",
 "resp_time": "0.512",
 "upstream_addr": "X:8000, Y:8000",
 "upstream_status": "502, 204",
 "traceparent": "00-0bc72f67dd2ee53ce6d7f03e4a4eb7d6-3a5171c169e8eb46-01",
 "trace_id": "0bc72f67dd2ee53ce6d7f03e4a4eb7d6",
 "span_id": "3a5171c169e8eb46",
 "org_slug": "",
 "matched_uri": ""
   }
   ```
   
   Sorry — I don’t have more relevant logs or additional steps to reproduce. 
This seems like a corner case and may not be easy to reproduce reliably.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]

2026-02-01 Thread via GitHub 


AlinsRan commented on issue #2708:
URL: 
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3832576013

   Thank you for reporting this issue and for the details you’ve provided.
   
   We attempted to reproduce the problem in our local environment using the 
following steps:
   
   > 1. deploy apisix and apisix-ingress-controller:
   > 
   > ```shell
   > helm install apisix \
   >   --namespace ingress-apisix \
   >   --create-namespace \
   >   --set apisix.deployment.role=traditional \
   >   --set apisix.deployment.role_traditional.config_provider=yaml \
   >   --set etcd.enabled=false \
   >   --set ingress-controller.enabled=true \
   >   --set ingress-controller.config.provider.type=apisix-standalone \
   >   --set ingress-controller.apisix.adminService.namespace=ingress-apisix \
   >   --set ingress-controller.gatewayProxy.createDefault=true \
   >   apisix/apisix
   > ```
   > 
   > 2. deploy httpbin upstream:
   > 
   > ```shell
   > kubectl apply -f 
https://raw.githubusercontent.com/apache/apisix-ingress-controller/refs/heads/v2.0.0/examples/httpbin/deployment.yaml
   > ```
   > 
   > 3. apply route
   > 
   > ```yaml
   > apiVersion: apisix.apache.org/v2
   > kind: ApisixRoute
   > metadata:
   >   name: getting-started-ip
   > spec:
   >   ingressClassName: apisix
   >   http:
   > - name: getting-started-ip
   >   match:
   > paths:
   >   - /ip
   >   backends:
   > - serviceName: httpbin
   >   servicePort: 80
   > ```
   > 
   > 4. scale httpbin to 4 replicas:
   > 
   > ```shell
   > kubectl scale deployment httpbin-deployment --replicas=4
   > ```
   > 
   > Check the upstream configuration of the data plane:
   > 
   > ```shell
   > curl localhost:9180/apisix/admin/configs -H "X-API-Key: 
edd1c9f034335f136f87ad84b625c8f1" -s | jq .upstreams
   > [
   >   {
   > "id": "ad735c2f",
   > "type": "roundrobin",
   > "labels": {
   >   "managed-by": "apisix-ingress-controller"
   > },
   > "name": "default_getting-started-ip_0",
   > "nodes": [
   >   {
   > "port": 80,
   > "host": "10.244.0.7",
   > "weight": 100
   >   },
   >   {
   > "port": 80,
   > "host": "10.244.0.15",
   > "weight": 100
   >   },
   >   {
   > "port": 80,
   > "host": "10.244.0.14",
   > "weight": 100
   >   },
   >   {
   > "port": 80,
   > "host": "10.244.0.13",
   > "weight": 100
   >   }
   > ],
   > "modifiedIndex": 1769998251570
   >   }
   > ]
   > ```
   > 
   > Check the logs after sending the request:
   > 
   > https://github.com/user-attachments/assets/b0b96c65-44e5-43a0-806d-2751c590da2a";
 />
   > 
   > 
   > 5. scale httpbin to 1 replicas:
   > 
   > ```shell
   > kubectl scale deployment httpbin-deployment --replicas=1
   > ```
   > 
   > Check the upstream configuration of the data plane:
   > 
   > ```shell
   > curl localhost:9180/apisix/admin/configs -H "X-API-Key: 
edd1c9f034335f136f87ad84b625c8f1" -s | jq .upstreams
   > [
   >   {
   > "id": "ad735c2f",
   > "type": "roundrobin",
   > "labels": {
   >   "managed-by": "apisix-ingress-controller"
   > },
   > "name": "default_getting-started-ip_0",
   > "nodes": [
   >   {
   > "port": 80,
   > "host": "10.244.0.7",
   > "weight": 100
   >   }
   > ],
   > "modifiedIndex": 1769998390864
   >   }
   > ]
   > ```
   > 
   > Check the logs after sending the request:
   > 
   > https://github.com/user-attachments/assets/9411d88c-656a-4662-b8ec-11aa250495d8";
 />
   
   
   However, under these conditions, we were **unable to reproduce the issue**.
   
   To help us investigate further, could you please provide some additional 
information?
   
   ### Minimal Reproducible Example
   - A simplified configuration or setup that can reliably reproduce the issue
   - If applicable, a sample request along with the expected and actual behavior
   
   ### Logs or Error Output
   - Relevant error or warning logs
   - Stack traces or additional diagnostic output, if available
   
   ### Additional Context
   - Whether the issue occurs consistently or intermittently
   - Any specific conditions under which the issue is more likely to occur
   
   With this information, we should be able to narrow down the root cause more 
effectively.  
   Thanks again for your report, and we appreciate your help in moving this 
forward.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]