[
https://issues.apache.org/jira/browse/FLINK-41035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41035:
-----------------------------------
Labels: pull-request-available (was: )
> Operator health probe reports "All informers are unhealthy" when a single
> event source fails with stop-on-informer-error=false
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41035
> URL: https://issues.apache.org/jira/browse/FLINK-41035
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
>
> h1. Problem
> With {{{}kubernetes.operator.startup.stop-on-informer-error=false{}}}, the
> operator is supposed to start and keep running even if some informers cannot
> be started, for example because of missing RBAC permissions. The operator
> health probe tolerates informers that were already unhealthy at startup, and
> only reports unhealthy if an informer becomes unhealthy later, or if all
> informers are unhealthy.
> The "all informers are unhealthy" check does not actually look at all
> informers. Whether it triggers depends on how many namespaces an event source
> covers, not on the real state of the operator:
> * *One watched namespace, {{Ingress}} cannot be watched:* the probe logs
> {{All informers are unhealthy}} and returns 500, although every other
> informer (FlinkDeployment, Deployment, FlinkSessionJob, FlinkStateSnapshot,
> ...) is healthy. The startup probe then kills the operator, and this repeats
> on every start.
> * *Two watched namespaces, {{Ingress}} cannot be watched in one of them:*
> the probe returns 200, and the failing informers are tolerated as expected.
> The failure is the same in both cases (one resource type cannot be watched in
> one namespace), but the outcome is opposite. In the first case
> {{stop-on-informer-error=false}} has no effect, because the operator never
> stays up.
> h1. Root cause
> {{InformerHealthSummary.fromRuntimeInfo}} computes "is any informer healthy"
> from
> {{{}RuntimeInfo.unhealthyInformerWrappingEventSourceHealthIndicator(){}}}.
> That JOSDK view only returns the event sources that contain at least one
> unhealthy informer. Fully healthy event sources are filtered out.
> As a result, a healthy informer is only seen when it shares an event source
> with an unhealthy one, which in practice means the same resource type in
> another watched namespace. With a single watched namespace (or when watching
> all namespaces), every event source has exactly one informer, so the check
> only ever sees the failing informers and concludes that all informers are
> unhealthy. This check runs before the startup tolerance, so the tolerated
> informers never get a chance to be accepted.
> h1. How to reproduce
> h2. Setup
> * Operator installed with the Helm chart, watching only the {{default}}
> namespace ({{{}watchNamespaces: [default]{}}}), namespace-scoped RBAC.
> * {{kubernetes.operator.startup.stop-on-informer-error: false}}
> * {{kubernetes.operator.ingress.manage}} left at its default ({{{}true{}}}),
> so the FlinkDeployment and FlinkBlueGreenDeployment controllers register an
> {{Ingress}} informer.
> h2. Steps
> # Remove the ingress permission from the operator Role:
> {code:bash}
> kubectl -n default get role flink-operator -o json \
> | jq 'del(.rules[] | select(.apiGroups == ["networking.k8s.io"]))' \
> | kubectl replace -f -
> {code}
> # Restart the operator:
> {code:bash}
> kubectl rollout restart deploy/flink-kubernetes-operator
> {code}
> h2. Observed
> After the cache sync timeout, the two {{Ingress}} informers are tolerated as
> initially unhealthy, the health endpoint starts, and the first probe already
> fails:
> {noformat}
> 2026-10-19 18:23:12,211 WARN [ingresses.networking.k8s.io/v1]
> i.j.o.p.e.s.informer.InformerWrapper Informer startup error. Will
> periodically retry. Informer:
> networking.k8s.io/v1/namespaces/default/ingresses
> 2026-10-19 18:23:12,213 WARN [ingresses.networking.k8s.io/v1]
> i.j.o.p.e.s.informer.InformerWrapper Informer startup error. Will
> periodically retry. Informer:
> networking.k8s.io/v1/namespaces/default/ingresses
> 2026-10-19 18:23:12,217 INFO main o.a.f.k.operator.health.HealthProbe
> Initially unhealthy informers:
> [InformerIdentifier(controllerName=flinkbluegreendeploymentcontroller,
> eventSourceName=...InformerEventSource@4893b344, informerName=default),
> InformerIdentifier(controllerName=flinkdeploymentcontroller,
> eventSourceName=...InformerEventSource@20411320, informerName=default)]
> 2026-10-19 18:23:12,270 INFO main o.a.f.k.o.h.OperatorHealthService Health
> probe HTTP endpoint started on port 8085
> 2026-10-19 18:23:20,954 ERROR nioEventLoopGroup-3-1
> o.a.f.k.operator.health.HealthProbe All informers are unhealthy
> {noformat}
> {noformat}
> Warning Unhealthy kubelet Startup probe failed: HTTP probe failed with
> statuscode: 500
> {noformat}
> The operator container is restarted once the startup probe budget is
> exhausted, and the same sequence repeats on every start.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
