Re: [I] bug: AIC broken until restart after failover test [apisix-ingress-controller]
boscard commented on issue #2734: URL: https://github.com/apache/apisix-ingress-controller/issues/2734#issuecomment-4486202864 Do you have any updates on it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: AIC broken until restart after failover test [apisix-ingress-controller]
Baoyuantop commented on issue #2734: URL: https://github.com/apache/apisix-ingress-controller/issues/2734#issuecomment-4155011318 Thank you for providing the information. We will conduct further investigation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: AIC broken until restart after failover test [apisix-ingress-controller]
martin-schulze-e2m commented on issue #2734: URL: https://github.com/apache/apisix-ingress-controller/issues/2734#issuecomment-4141572982 Sorry for the late response. I had another look at our logs but I am afraid they are rather patchy due to the failover as well. This is the output of a `kubectl get events --follow` job around the time of the failover: ``` 2026-03-19 15:47:59.978 apisix35mNormal LeaderElection Lease/apisix-ingress-controller-leader apisix-ingress-controller-6788f78f4c-gd95r_6c70e4c1-7a96-43de-9cea-8dd53ae2202a became leader 2026-03-19 15:47:59.977 apisix38m Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Readiness probe failed: Get "http://10.62.17.29:8081/readyz": read tcp 10.62.17.1:52512->10.62.17.29:8081: read: connection reset by peer 2026-03-19 15:47:59.977 apisix38mNormal Killing Pod/apisix-ingress-controller-6788f78f4c-gd95r Container manager failed liveness probe, will be restarted 2026-03-19 15:47:59.976 apisix38m (x4 over 39m) Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Liveness probe failed: Get "http://10.62.17.29:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-03-19 15:47:59.976 apisix37m (x13 over 39m) Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Readiness probe failed: Get "http://10.62.17.29:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-03-19 15:47:59.975 apisix38m (x2 over 39m) Normal Started Pod/apisix-ingress-controller-6788f78f4c-gd95r Started container manager 2026-03-19 15:47:59.974 apisix38m (x2 over 39m) Normal Created Pod/apisix-ingress-controller-6788f78f4c-gd95r Created container: manager 2026-03-19 15:47:59.974 apisix38m (x2 over 39m) Normal Pulled Pod/apisix-ingress-controller-6788f78f4c-gd95r Container image "apache/apisix-ingress-controller:2.0.1" already present on machine 2026-03-19 15:47:59.973 apisix35mNormal EnsuredLoadBalancer Service/apisix-gateway Ensured load balancer 2026-03-19 15:47:59.972 apisix35mNormal EnsuringLoadBalancer Service/apisix-gateway Ensuring load balancer 2026-03-19 15:13:15.845 apisix18s NormalLeaderElection Lease/apisix-ingress-controller-leader apisix-ingress-controller-6788f78f4c-gd95r_6c70e4c1-7a96-43de-9cea-8dd53ae2202a became leader 2026-03-19 15:13:15.844 apisix3m38s Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Readiness probe failed: Get "http://10.62.17.29:8081/readyz": read tcp 10.62.17.1:52512->10.62.17.29:8081: read: connection reset by peer 2026-03-19 15:13:15.843 apisix3m38s NormalKilling Pod/apisix-ingress-controller-6788f78f4c-gd95r Container manager failed liveness probe, will be restarted 2026-03-19 15:13:15.842 apisix3m18s (x4 over 4m18s) Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Liveness probe failed: Get "http://10.62.17.29:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-03-19 15:13:15.841 apisix2m58s (x13 over 4m38s) Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Readiness probe failed: Get "http://10.62.17.29:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-03-19 15:13:15.840 apisix3m37s (x2 over 4m51s) NormalStarted Pod/apisix-ingress-controller-6788f78f4c-gd95r Started container manager 2026-03-19 15:13:15.821 apisix3m37s (x2 over 4m51s) NormalCreated Pod/apisix-ingress-controller-6788f78f4c-gd95r Created container: manager 2026-03-19 15:13:15.789 apisix3m37s (x2 over 4m51s) NormalPulled Pod/apisix-ingress-controller-6788f78f4c-gd95r Container image "apache/apisix-ingress-controller:2.0.1" already present on machine 2026-03-19 15:13:15.789 apisix23sNormal EnsuredLoadBalancer Service/apisix-gatewayEnsured load balancer 2026-03-19 15:13:15.788 apisix23sNormal EnsuringLoadBalancer Service/apisix-
Re: [I] bug: AIC broken until restart after failover test [apisix-ingress-controller]
Baoyuantop commented on issue #2734: URL: https://github.com/apache/apisix-ingress-controller/issues/2734#issuecomment-4107741309 Thanks for the detailed logs. From the log timeline, the failure window shows kube-apiserver connection failures, leader election loss, and API discovery failures, followed by persistent `Index with name field:XXX does not exist` errors. Based on the code, AIC checks API resource availability via discovery during startup to decide whether to register field indexes. If the API server is unreachable at that point, index registration gets skipped. Once the cluster recovers, the indexes are never retroactively registered, causing continuous Reconcile failures until restart. To help narrow down the root cause further, could you provide: 1. Whether the ingress-controller pod was recreated during/after failover (and if so, whether the API server was still unstable at that time) 2. Full startup logs showing `api-detection` / `Skipping indexer setup` lines 3. Whether any CRD/APIService availability changed during failover 4. A minimal reproduction procedure if possible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
