Dennis-Mircea Ciupitu created FLINK-41465:
---------------------------------------------
Summary: Health probe reports healthy while the informer receives
no events
Key: FLINK-41465
URL: https://issues.apache.org/jira/browse/FLINK-41465
Project: Flink
Issue Type: Bug
Components: Kubernetes Operator
Reporter: Dennis-Mircea Ciupitu
h1. Problem
The operator learns about resources only through the watch connections of its
informers. The health probe judges an informer by JOSDK's health indicator,
which comes down to {{isRunning()}} and {{{}hasSynced(){}}}: the informer was
started and completed its first list. Neither says anything about whether
events still arrive.
Two failure classes leave an informer silent while the probe keeps answering
200. The operator then ignores new resources and spec changes until someone
restarts it. Existing deployments are still reconciled on the timer, so nothing
else looks wrong.
h2. 1. The watch is alive and delivers nothing
Seen in production on operator 1.16.1: a FlinkDeployment created in a watched
namespace had no status, no event and no log line for 12 hours. The API server
returned it on every read and {{kubectl get --watch}} received its events, the
operator log showed no connection error at all, and the operator's watch
connections stayed open. This matches a stale watch cache on the API server
replica the operator's connections were pinned to. A restart fixed it
instantly, because the fresh informers listed everything.
The same state is reproducible with a TLS proxy between the operator and the
API server that forwards everything except the server-to-client data frames of
watch connections (events and bookmarks) while keeping ping, pong and close
frames. The watches stay connected, the pings are answered, and a
FlinkDeployment created behind the proxy never gets a status, with the health
endpoint at 200 on every check. The API server's own watch timeout does not
help: it closes each watch after ten minutes, fabric8 reopens it and the
replayed ADD event is lost again. A connection that becomes half-open, when the
server side disappears without a TCP reset, produces a shorter blind window of
one to two minutes with the same 200 answers.
A resource created during such a window also depends on the next watch working:
after a restart the first reconcile only adds the operator finalizer, and the
actual deployment is triggered by the watch event of that update.
h2. 2. The watch is down and keeps retrying
A watch that fails after startup is retried forever by fabric8:
{{{}InformerHealthIndicator.{}}}{{{}isRunning(){}}} and
{{{}InformerHealthIndicator.{}}}{{{}hasSynced(){}}} stay true, so JOSDK reports
the informer healthy.
This is reproducible by deleting a watched namespace on an operator installed
with namespace-scoped RBAC: the log shows an endless {{Received 403 on
websocket}} loop while the probe stays OK. The same holds for an API server
that becomes unreachable after startup. Only the deprecated JOSDK
{{InformerHealthIndicator.isWatching()}} would show it, and it is not part of
JOSDK's health status.
h1. Root cause
The probe evaluates the client's belief about its watch and the completion of
the initial sync. It never looks at the one thing that proves a watch is alive:
data arriving. The canary resources do look at it end to end, but only where a
canary is deployed, only for the FlinkDeployment informer of that namespace,
and at the cost of their own write traffic.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
