sopacifics opened a new issue, #2699: URL: https://github.com/apache/apisix-ingress-controller/issues/2699
### Current Behavior Similar issue -> https://github.com/apache/apisix-ingress-controller/issues/2689 Environment (testing in DEV): - APISIX Helm-chart v2.12.5 and AppVersion 3.14.1 - apisix-ingress-controller: 2.0.0 - APISIX: 3.14.1 - Kubernetes 1.33 After upgrading from 1.8.4 to 2.0.0, ingress controller no longer updates APISIX upstreams when pods restart with new IPs. - No errors in apisix-ingress-controller, controller logs show "syncing all resources" with "success_count" but APISIX admin API shows stale pod IPs in upstreams. - Results in 502 errors as traffic routes to dead pods. **Expected Behavior:** _When pods restart/scale and get new IPs, controller should detect endpoint changes and update corresponding upstreams in APISIX (as it did in 1.8.4)._ Steps to Reproduce: - Deploy ApisixRoute with backend service - Note upstream IPs in APISIX - Delete/Restart pod to force restart with new IP - Check APISIX upstream - still has old IP - Controller logs show no errors **NO Workaround** Regression: This worked correctly in 1.8.4 but this version doesn't seem compatible with new APISIX Helm-chart is that correct? Apisixroute example ``` apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: annotations: argocd.argoproj.io/tracking-id: cxp-query-service:apisix.apache.org/ApisixRoute:cxp-query-service/cxp-baqs labels: app.kubernetes.io/instance: cxp-query-service app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: cxp-query-service app.kubernetes.io/version: 1.16.0 helm.sh/chart: cxp-query-service-0.0.213 name: cxp-baqs namespace: cxp-query-service spec: http: - backends: - serviceName: cxp-baqs servicePort: 4000 match: methods: - GET paths: - /cxp-baqs/api/v1/graphql name: graphql-playground plugins: - config: enable_authorization: false enable: true name: oidc-with-iam - config: regex_uri: - ^/cxp-baqs/api/v1/(.*) - /$1 enable: true name: proxy-rewrite - config: count: 4 rejected_code: 503 time_window: 1 enable: true name: limit-count status: conditions: - message: Sync Successfully observedGeneration: 1 reason: ResourcesSynced status: 'True' type: ResourcesAvailable ``` Route works as expected, the issue is when a new IP is assigned to the POD Before restart ``` kubectl get pods # OUTPUT: cxp-baqs 1/1 Running 0 27m 10.13.147.219 ``` - Route return 200 After restart ``` kubectl get pods # OUTPUT: cxp-baqs 1/1 Running 0 27m 10.13.163.151 ``` Error ``` # ERRORS: 2026/01/08 23:11:37 [error] 52#52: *71860 connect() failed (113: No route to host) while connecting to upstream, client: 10.13.204.241, server: _, request: "GET /cxp-baqs/api/v1 HTTP/1.1", upstream: "http://10.13.147.219:8080/api/scopes/default";, ← DEAD IP host: "dev.cxp.cisco.com" ``` ``` # From APISIX access logs: 10.13.185.224 - - [08/Jan/2026:23:39:32 +0000] dev.cxp.cisco.com "GET /ui HTTP/1.1" 502 631 9.327 "-" "Mozilla/5.0..." 10.13.147.219:8080 ← Trying this IP 502 ← All failed 3.104 ← 3 seconds per attempt ``` ``` kubectl logs -n apisix deployment/apisix-ingress-controller --tail=50 # OUTPUT: 2026-01-08T23:34:37.449Z INFO provider.client syncing all resources 2026-01-08T23:34:37.449Z INFO provider.client syncing resources for config {"service_number": 16} 2026-01-08T23:34:47.376Z INFO provider.client syncing all resources 2026-01-08T23:34:47.376Z INFO provider.client syncing resources for config {"service_number": 16} ``` Can you please help with this? ### Expected Behavior _No response_ ### Error Logs _No response_ ### Steps to Reproduce Steps to Reproduce: - Deploy ApisixRoute with backend service - Note upstream IPs in APISIX - Delete/Restart pod to force restart with new IP - Check APISIX upstream - still has old IP - Controller logs show no errors ### Environment Environment (testing in DEV): - APISIX Helm-chart v2.12.5 and AppVersion 3.14.1 - apisix-ingress-controller: 2.0.0 - APISIX: 3.14.1 - Kubernetes 1.33 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
