martin-schulze-e2m commented on issue #2734: URL: https://github.com/apache/apisix-ingress-controller/issues/2734#issuecomment-4141572982
Sorry for the late response. I had another look at our logs but I am afraid they are rather patchy due to the failover as well. This is the output of a `kubectl get events --follow` job around the time of the failover: ``` 2026-03-19 15:47:59.978 apisix 35m Normal LeaderElection Lease/apisix-ingress-controller-leader apisix-ingress-controller-6788f78f4c-gd95r_6c70e4c1-7a96-43de-9cea-8dd53ae2202a became leader 2026-03-19 15:47:59.977 apisix 38m Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Readiness probe failed: Get "http://10.62.17.29:8081/readyz": read tcp 10.62.17.1:52512->10.62.17.29:8081: read: connection reset by peer 2026-03-19 15:47:59.977 apisix 38m Normal Killing Pod/apisix-ingress-controller-6788f78f4c-gd95r Container manager failed liveness probe, will be restarted 2026-03-19 15:47:59.976 apisix 38m (x4 over 39m) Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Liveness probe failed: Get "http://10.62.17.29:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-03-19 15:47:59.976 apisix 37m (x13 over 39m) Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Readiness probe failed: Get "http://10.62.17.29:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-03-19 15:47:59.975 apisix 38m (x2 over 39m) Normal Started Pod/apisix-ingress-controller-6788f78f4c-gd95r Started container manager 2026-03-19 15:47:59.974 apisix 38m (x2 over 39m) Normal Created Pod/apisix-ingress-controller-6788f78f4c-gd95r Created container: manager 2026-03-19 15:47:59.974 apisix 38m (x2 over 39m) Normal Pulled Pod/apisix-ingress-controller-6788f78f4c-gd95r Container image "apache/apisix-ingress-controller:2.0.1" already present on machine 2026-03-19 15:47:59.973 apisix 35m Normal EnsuredLoadBalancer Service/apisix-gateway Ensured load balancer 2026-03-19 15:47:59.972 apisix 35m Normal EnsuringLoadBalancer Service/apisix-gateway Ensuring load balancer 2026-03-19 15:13:15.845 apisix 18s Normal LeaderElection Lease/apisix-ingress-controller-leader apisix-ingress-controller-6788f78f4c-gd95r_6c70e4c1-7a96-43de-9cea-8dd53ae2202a became leader 2026-03-19 15:13:15.844 apisix 3m38s Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Readiness probe failed: Get "http://10.62.17.29:8081/readyz": read tcp 10.62.17.1:52512->10.62.17.29:8081: read: connection reset by peer 2026-03-19 15:13:15.843 apisix 3m38s Normal Killing Pod/apisix-ingress-controller-6788f78f4c-gd95r Container manager failed liveness probe, will be restarted 2026-03-19 15:13:15.842 apisix 3m18s (x4 over 4m18s) Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Liveness probe failed: Get "http://10.62.17.29:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-03-19 15:13:15.841 apisix 2m58s (x13 over 4m38s) Warning Unhealthy Pod/apisix-ingress-controller-6788f78f4c-gd95r Readiness probe failed: Get "http://10.62.17.29:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 2026-03-19 15:13:15.840 apisix 3m37s (x2 over 4m51s) Normal Started Pod/apisix-ingress-controller-6788f78f4c-gd95r Started container manager 2026-03-19 15:13:15.821 apisix 3m37s (x2 over 4m51s) Normal Created Pod/apisix-ingress-controller-6788f78f4c-gd95r Created container: manager 2026-03-19 15:13:15.789 apisix 3m37s (x2 over 4m51s) Normal Pulled Pod/apisix-ingress-controller-6788f78f4c-gd95r Container image "apache/apisix-ingress-controller:2.0.1" already present on machine 2026-03-19 15:13:15.789 apisix 23s Normal EnsuredLoadBalancer Service/apisix-gateway Ensured load balancer 2026-03-19 15:13:15.788 apisix 23s Normal EnsuringLoadBalancer Service/apisix-gateway Ensuring load balancer ``` Unfortunately, it seems the events on CRDs are not displayed (or there were none). So to answer your questions: 1. our kube_pod_container_status_restarts_total metric says pod apisix-ingress-controller-6788f78f4c-gd95 restarted once at 15:13 I can't say for sure if the API was down at this time but I'd assume as much. 2. here are more logs from the ingress controller, this time excluding error messages: ``` 2026-03-19T15:18:25.204Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:17:57.300Z INFO provider.client client/client.go:182 no GatewayProxy configs provided 2026-03-19T15:17:57.300Z INFO provider.client client/client.go:177 syncing all resources 2026-03-19T15:16:57.301Z INFO provider.client client/client.go:182 no GatewayProxy configs provided 2026-03-19T15:16:57.300Z INFO provider.client client/client.go:177 syncing all resources 2026-03-19T15:15:57.300Z INFO provider.client client/client.go:182 no GatewayProxy configs provided 2026-03-19T15:15:57.300Z INFO provider.client client/client.go:177 syncing all resources 2026-03-19T15:15:41.364Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:14:57.301Z INFO provider.client client/client.go:182 no GatewayProxy configs provided 2026-03-19T15:14:57.301Z INFO provider.client client/client.go:177 syncing all resources 2026-03-19T15:14:19.442Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:13:57.301Z INFO provider.client client/client.go:182 no GatewayProxy configs provided 2026-03-19T15:13:57.301Z INFO provider.client client/client.go:177 syncing all resources 2026-03-19T15:13:38.481Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:13:18.000Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:13:07.759Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:13:02.638Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:13:00.077Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:58.796Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:58.156Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:57.835Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:57.674Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:57.594Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:57.553Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:57.532Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:57.522Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:57.517Z DEBUG controllers.GatewayProxy controller/utils.go:1473 to process provider endpoints by provider.service {"service": {"name":"apisix-admin","namespace":"apisix"}} 2026-03-19T15:12:57.516Z INFO controller-runtime controller/controller.go:248 Starting workers {"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "worker count": 1} 2026-03-19T15:12:57.516Z INFO controller-runtime controller/controller.go:239 Starting Controller {"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy"} 2026-03-19T15:12:57.300Z INFO controller-runtime controller/controller.go:204 Starting EventSource {"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1alpha1.GatewayProxy"} 2026-03-19T15:12:57.300Z INFO controller-runtime controller/controller.go:204 Starting EventSource {"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1.Secret"} 2026-03-19T15:12:57.300Z INFO controller-runtime controller/controller.go:204 Starting EventSource {"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1.Service"} 2026-03-19T15:12:57.300Z INFO controller-runtime controller/controller.go:204 Starting EventSource {"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1.EndpointSlice"} 2026-03-19T15:12:57.300Z INFO controller-runtime controller/controller.go:204 Starting EventSource {"controller": "gatewayproxy", "controllerGroup": "apisix.apache.org", "controllerKind": "GatewayProxy", "source": "kind source: *v1.IngressClass"} 2026-03-19T15:12:57.300Z INFO status.updater status/updater.go:131 started status update handler 2026-03-19T15:12:57.300Z INFO provider apisix/provider.go:256 Ready detected, starting sync loop 2026-03-19T15:12:57.300Z INFO provider apisix/provider.go:254 starting provider, waiting for readiness 2026-03-19T15:12:57.300Z INFO readiness readiness/manager.go:139 readiness manager started 2026-03-19T15:12:57.300Z DEBUG controller-runtime.events recorder/recorder.go:104 apisix-ingress-controller-6788f78f4c-gd95r_6c70e4c1-7a96-43de-9cea-8dd53ae2202a became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"apisix","name":"apisix-ingress-controller-leader","uid":"769a49a6-01d2-42c5-abcf-64a92892c597","apiVersion":"coordination.k8s.io/v1","resourceVersion":"247335807"}, "reason": "LeaderElection"} I0319 15:12:57.300009 1 leaderelection.go:271] successfully acquired lease apisix/apisix-ingress-controller-leader I0319 15:11:52.703743 1 leaderelection.go:257] attempting to acquire leader lease apisix/apisix-ingress-controller-leader... 2026-03-19T15:11:52.703Z INFO controller-runtime.controller-runtime.certwatcher certwatcher/certwatcher.go:133 Starting certificate poll+watcher {"interval": "10s"} 2026-03-19T15:11:52.703Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:242 Serving webhook server {"host": "", "port": 9443} 2026-03-19T15:11:52.703Z INFO controller-runtime.controller-runtime.certwatcher certwatcher/certwatcher.go:211 Updated current TLS certificate 2026-03-19T15:11:52.703Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:191 Starting webhook server 2026-03-19T15:11:52.703Z INFO controller-runtime.controller-runtime.metrics server/server.go:247 Serving metrics server {"bindAddress": ":8080", "secure": false} 2026-03-19T15:11:52.703Z INFO controller-runtime manager/server.go:83 starting server {"name": "health probe", "addr": "[::]:8081"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.metrics server/server.go:208 Starting metrics server 2026-03-19T15:11:52.702Z INFO setup manager/run.go:264 starting controller manager 2026-03-19T15:11:52.702Z INFO setup manager/run.go:258 setting up ready checks 2026-03-19T15:11:52.702Z INFO setup manager/run.go:252 setting up health checks 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-apisix-apache-org-v1alpha1-consumer"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "apisix.apache.org/v1alpha1, Kind=Consumer", "path": "/validate-apisix-apache-org-v1alpha1-consumer"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-apisix-apache-org-v2-apisixroute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "apisix.apache.org/v2, Kind=ApisixRoute", "path": "/validate-apisix-apache-org-v2-apisixroute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-apisix-apache-org-v2-apisixtls"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "apisix.apache.org/v2, Kind=ApisixTls", "path": "/validate-apisix-apache-org-v2-apisixtls"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-apisix-apache-org-v2-apisixconsumer"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "apisix.apache.org/v2, Kind=ApisixConsumer", "path": "/validate-apisix-apache-org-v2-apisixconsumer"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-gateway-networking-k8s-io-v1alpha2-udproute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute", "path": "/validate-gateway-networking-k8s-io-v1alpha2-udproute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-gateway-networking-k8s-io-v1alpha2-tcproute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute", "path": "/validate-gateway-networking-k8s-io-v1alpha2-tcproute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-gateway-networking-k8s-io-v1-grpcroute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "gateway.networking.k8s.io/v1, Kind=GRPCRoute", "path": "/validate-gateway-networking-k8s-io-v1-grpcroute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-gateway-networking-k8s-io-v1-httproute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "gateway.networking.k8s.io/v1, Kind=HTTPRoute", "path": "/validate-gateway-networking-k8s-io-v1-httproute"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-apisix-apache-org-v1alpha1-gatewayproxy"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "apisix.apache.org/v1alpha1, Kind=GatewayProxy", "path": "/validate-apisix-apache-org-v1alpha1-gatewayproxy"} 2026-03-19T15:11:52.702Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-gateway-networking-k8s-io-v1-gateway"} 2026-03-19T15:11:52.701Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "gateway.networking.k8s.io/v1, Kind=Gateway", "path": "/validate-gateway-networking-k8s-io-v1-gateway"} 2026-03-19T15:11:52.701Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-networking-k8s-io-v1-ingressclass"} 2026-03-19T15:11:52.701Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "networking.k8s.io/v1, Kind=IngressClass", "path": "/validate-networking-k8s-io-v1-ingressclass"} 2026-03-19T15:11:52.701Z INFO controller-runtime.controller-runtime.webhook webhook/server.go:183 Registering webhook {"path": "/validate-networking-k8s-io-v1-ingress"} 2026-03-19T15:11:52.700Z INFO controller-runtime.controller-runtime.builder builder/webhook.go:226 Registering a validating webhook {"GVK": "networking.k8s.io/v1, Kind=Ingress", "path": "/validate-networking-k8s-io-v1-ingress"} 2026-03-19T15:11:52.700Z INFO setup manager/run.go:243 setting up webhooks 2026-03-19T15:11:52.700Z INFO setup.apiv2 manager/controllers.go:291 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixGlobalRule"} 2026-03-19T15:11:52.699Z INFO setup.apiv2 manager/controllers.go:291 Skipping indexer setup, API not found in cluster {"api": "networking.k8s.io/v1, Kind=Ingress"} 2026-03-19T15:11:52.699Z INFO setup.apiv2 manager/controllers.go:291 Skipping indexer setup, API not found in cluster {"api": "networking.k8s.io/v1, Kind=IngressClass"} 2026-03-19T15:11:52.699Z INFO setup.apiv2 manager/controllers.go:291 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixUpstream"} 2026-03-19T15:11:52.698Z INFO setup.apiv2 manager/controllers.go:291 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixTls"} 2026-03-19T15:11:52.698Z INFO setup.apiv2 manager/controllers.go:291 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixPluginConfig"} 2026-03-19T15:11:52.697Z INFO setup.apiv2 manager/controllers.go:291 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixConsumer"} 2026-03-19T15:11:52.697Z INFO setup.apiv2 manager/controllers.go:291 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixRoute"} 2026-03-19T15:11:52.695Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixPluginConfig"} 2026-03-19T15:11:52.695Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixRoute"} 2026-03-19T15:11:52.694Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixConsumer"} 2026-03-19T15:11:52.694Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "networking.k8s.io/v1, Kind=Ingress"} 2026-03-19T15:11:52.694Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "networking.k8s.io/v1, Kind=IngressClass"} 2026-03-19T15:11:52.693Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixGlobalRule"} 2026-03-19T15:11:52.693Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixTls"} 2026-03-19T15:11:52.692Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"} 2026-03-19T15:11:52.692Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"} 2026-03-19T15:11:52.692Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"} 2026-03-19T15:11:52.691Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"} 2026-03-19T15:11:52.691Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"} 2026-03-19T15:11:52.690Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=Gateway"} 2026-03-19T15:11:52.690Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"} 2026-03-19T15:11:52.689Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=Consumer"} 2026-03-19T15:11:52.689Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"} 2026-03-19T15:11:52.688Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"} 2026-03-19T15:11:52.688Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"} 2026-03-19T15:11:52.687Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"} 2026-03-19T15:11:52.687Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=Gateway"} 2026-03-19T15:11:52.686Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"} 2026-03-19T15:11:52.685Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"} 2026-03-19T15:11:52.685Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=Consumer"} 2026-03-19T15:11:52.684Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=BackendTrafficPolicy"} 2026-03-19T15:11:52.684Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=GatewayProxy"} 2026-03-19T15:11:52.683Z INFO setup manager/run.go:227 setting up controllers 2026-03-19T15:11:52.683Z INFO setup manager/run.go:223 CRD ReferenceGrants is not installed {"err": "failed to get server groups: Get \"https://10.63.0.1:443/api\": dial tcp 10.63.0.1:443: connect: connection refused"} 2026-03-19T15:11:52.683Z INFO setup manager/run.go:216 check ReferenceGrants is enabled 2026-03-19T15:11:52.683Z INFO provider.client client/client.go:63 ADC client initialized 2026-03-19T15:11:52.683Z INFO controller-runtime.readiness manager/controllers.go:384 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v1alpha1, Kind=Consumer"} 2026-03-19T15:11:52.682Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"} 2026-03-19T15:11:52.681Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"} 2026-03-19T15:11:51.661Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"} 2026-03-19T15:11:48.585Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"} 2026-03-19T15:11:45.515Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"} 2026-03-19T15:11:45.514Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixUpstream"} 2026-03-19T15:11:45.514Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixConsumer"} 2026-03-19T15:11:45.513Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixTls"} 2026-03-19T15:11:42.441Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixPluginConfig"} 2026-03-19T15:11:39.370Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixGlobalRule"} 2026-03-19T15:11:39.370Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixRoute"} 2026-03-19T15:11:39.369Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "networking.k8s.io/v1, Kind=Ingress"} 2026-03-19T15:11:38.173Z INFO root/root.go:125 controller start configuration {"config": {"log_level":"debug","controller_name":"apisix.apache.org/apisix-ingress-controller","leader_election_id":"apisix-ingress-controller-leader","metrics_addr":":8080","server_addr":":9092","enable_server":false,"enable_http2":false,"probe_addr":":8081","secure_metrics":false,"leader_election":{"lease_duration":"30s","renew_deadline":"20s","retry_period":"2s"},"exec_adc_timeout":"15s","provider":{"type":"apisix-standalone","sync_period":"1m0s","init_sync_delay":"20m0s"},"webhook":{"enable":true,"tls_cert_file":"tls.crt","tls_key_file":"tls.key","tls_cert_dir":"/certs","port":9443},"disable_gateway_api":false}} 2026-03-19T15:11:36.299Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixTls"} 2026-03-19T15:11:36.298Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixPluginConfig"} 2026-03-19T15:11:36.298Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixRoute"} 2026-03-19T15:11:36.297Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v2, Kind=ApisixConsumer"} 2026-03-19T15:11:35.279Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "networking.k8s.io/v1, Kind=Ingress"} 2026-03-19T15:11:35.278Z INFO controller-runtime.indexer.apiv2 indexer/indexer.go:93 Skipping indexer setup, API not found in cluster {"api": "networking.k8s.io/v1, Kind=IngressClass"} 2026-03-19T15:11:35.278Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"} 2026-03-19T15:11:35.277Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=Consumer"} 2026-03-19T15:11:33.227Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"} 2026-03-19T15:11:33.226Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"} 2026-03-19T15:11:33.225Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"} 2026-03-19T15:11:32.909Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"} 2026-03-19T15:11:29.833Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"} 2026-03-19T15:11:26.761Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=Gateway"} 2026-03-19T15:11:24.841Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"} 2026-03-19T15:11:21.769Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"} 2026-03-19T15:11:18.698Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"} 2026-03-19T15:11:18.697Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"} 2026-03-19T15:11:15.626Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"} 2026-03-19T15:11:15.625Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=Gateway"} 2026-03-19T15:11:12.553Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"} 2026-03-19T15:11:09.485Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=GatewayProxy"} 2026-03-19T15:11:06.411Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=Consumer"} 2026-03-19T15:11:06.410Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=BackendTrafficPolicy"} 2026-03-19T15:11:06.409Z INFO setup manager/run.go:227 setting up controllers 2026-03-19T15:11:06.409Z INFO setup manager/run.go:223 CRD ReferenceGrants is not installed {"err": "failed to get server groups: Get \"https://10.63.0.1:443/api\": dial tcp 10.63.0.1:443: connect: no route to host"} 2026-03-19T15:11:04.745Z INFO setup manager/run.go:216 check ReferenceGrants is enabled 2026-03-19T15:11:04.745Z INFO provider.client client/client.go:63 ADC client initialized 2026-03-19T15:11:04.745Z INFO controller-runtime.readiness manager/controllers.go:384 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v1alpha1, Kind=Consumer"} 2026-03-19T15:11:03.337Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"} 2026-03-19T15:11:01.673Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"} 2026-03-19T15:10:59.785Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"} 2026-03-19T15:10:58.601Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"} 2026-03-19T15:10:57.097Z INFO controller-runtime.readiness manager/controllers.go:324 Skipping readiness registration, API not found {"gvk": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"} 2026-03-19T15:10:55.113Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixUpstream"} 2026-03-19T15:10:52.041Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixConsumer"} 2026-03-19T15:10:50.729Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixTls"} 2026-03-19T15:10:48.777Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixPluginConfig"} 2026-03-19T15:10:47.657Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixGlobalRule"} 2026-03-19T15:10:44.585Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "apisix.apache.org/v2, Kind=ApisixRoute"} 2026-03-19T15:10:42.253Z INFO controller-runtime.readiness manager/controllers.go:353 Skipping readiness registration, API not found {"gvk": "networking.k8s.io/v1, Kind=Ingress"} 2026-03-19T15:10:38.185Z INFO root/root.go:125 controller start configuration {"config": {"log_level":"debug","controller_name":"apisix.apache.org/apisix-ingress-controller","leader_election_id":"apisix-ingress-controller-leader","metrics_addr":":8080","server_addr":":9092","enable_server":false,"enable_http2":false,"probe_addr":":8081","secure_metrics":false,"leader_election":{"lease_duration":"30s","renew_deadline":"20s","retry_period":"2s"},"exec_adc_timeout":"15s","provider":{"type":"apisix-standalone","sync_period":"1m0s","init_sync_delay":"20m0s"},"webhook":{"enable":true,"tls_cert_file":"tls.crt","tls_key_file":"tls.key","tls_cert_dir":"/certs","port":9443},"disable_gateway_api":false}} 2026-03-19T15:10:35.369Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"} 2026-03-19T15:10:32.297Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"} 2026-03-19T15:10:31.753Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"} 2026-03-19T15:10:29.229Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"} 2026-03-19T15:10:29.101Z INFO setup.gatewayapi manager/controllers.go:214 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=Gateway"} 2026-03-19T15:10:26.729Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=UDPRoute"} 2026-03-19T15:10:23.661Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TCPRoute"} 2026-03-19T15:10:22.957Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GRPCRoute"} 2026-03-19T15:10:20.585Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=HTTPRoute"} 2026-03-19T15:10:17.513Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=Gateway"} 2026-03-19T15:10:16.745Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1, Kind=GatewayClass"} 2026-03-19T15:10:13.673Z INFO controller-runtime.indexer.gatewayapi indexer/indexer.go:116 Skipping indexer setup, API not found in cluster {"api": "gateway.networking.k8s.io/v1alpha2, Kind=TLSRoute"} 2026-03-19T15:10:11.369Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=GatewayProxy"} 2026-03-19T15:10:10.541Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=Consumer"} 2026-03-19T15:10:07.469Z INFO controller-runtime.indexer.apiv1alpha1 indexer/indexer.go:71 Skipping indexer setup, API not found in cluster {"api": "apisix.apache.org/v1alpha1, Kind=BackendTrafficPolicy"} ``` 3. I am not sure how to check that. We had several services behind ApisixRoutes that restarted due to the failover. 4. I don't think this will be possible unless we plan to do another failover test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
