gada121982 opened a new issue, #862:
URL: https://github.com/apache/spark-kubernetes-operator/issues/862
## Summary
`KubernetesClientFactory` hard-wires fabric8's `VertxHttpClientFactory`.
fabric8's
Vert.x client ignores `Config#websocketPingInterval` (only the OkHttp client
applies
it), so the operator's watch WebSockets carry no traffic while the watched
resources are quiet. On a managed Kubernetes service whose API endpoint
closes
connections that are idle for 50 seconds, every informer watch is closed
every
~50s. The informers go `UNHEALTHY`, events are missed, and new applications
fail
with `DriverStartTimedOut` even though the driver pod was created and
started.
## Environment
- spark-kubernetes-operator 0.8.0 (fabric8 7.6.1). Also reproduced with 0.8.0
rebuilt against fabric8 7.8.0, the version 1.0.0 ships.
- Managed Kubernetes v1.30, API server reached through a provider load
balancer
that closes connections idle for 50s.
- `spark-operator/build.gradle` (0.8.0 and 1.0.0):
`implementation(libs.kubernetes.httpclient.vertx)`
## Symptoms
Operator log, repeating every ~50s for all informers at once:
```
DEBUG i.f.k.c.d.i.AbstractWatchManager Watching
https://10.96.0.1:443/api/v1/pods?allowWatchBookmarks=true&labelSelector=spark.operator%2Fname%3Dspark-kubernetes-operator&resourceVersion=...&timeoutSeconds=600&watch=true...
DEBUG i.f.k.c.d.i.WatcherWebSocketListener WebSocket successfully opened
... 50 seconds, no events ...
DEBUG i.f.k.c.d.i.AbstractWatchManager Watch connection error, will
reconnect if possible
java.io.IOException: io.vertx.core.http.HttpClosedException: Connection was
closed
INFO i.f.k.c.d.i.AbstractWatchManager Watch connection error received 335
times without progress, will reconnect if possible
ERROR o.a.s.k.o.p.HealthProbe Controller: sparkappreconciler, Event Source:
..., Informer: UNHEALTHY is in JOSDK_ALL_NAMESPACES, not a healthy state
```
The `resourceVersion` in the reconnect URL never advances: each watch lives
50s
without receiving a single event or bookmark (the API server sends bookmarks
roughly once a minute, i.e. later than the 50s idle cutoff).
Effect on applications:
```
KubernetesClientTimeoutException: Timed out waiting for [60000] milliseconds
for [Pod] with name:[<app>-0-driver] in namespace [<ns>].
ERROR ... SparkAppReconciler Failed attempt: 0, last attempt: false
WARN ... AppCleanUpStep Application is configured to restart, resources
created in current attempt would be force released.
```
and eventually the application ends in `DriverStartTimedOut`. The failure is
intermittent: it depends on whether the driver pod's events arrive while a
watch
is connected.
## Root cause
Measured from a pod on the same node, watching a ConfigMap that does not
change:
| Connection | Result |
|---|---|
| Chunked HTTP watch | closed after 50.0s |
| WebSocket watch, no pings (by Service IP or DNS name, with or without
fabric8's query parameters) | closed after 50.0s |
| WebSocket watch, client ping every 10s / 20s / 30s | alive for the whole
150s test |
| HTTP watch on a resource patched every 15s | alive for the whole 330s test
|
So the connection is dropped only when idle, and a client-side WebSocket ping
keeps it open. fabric8 defaults `websocketPingInterval` to 30s, but:
- `httpclient-okhttp` applies it (`OkHttpClientFactory`:
`pingInterval(...)`).
- `httpclient-vertx` does not reference it at all (checked 7.6.1, 7.8.0 and
`main`).
- `httpclient-jdk` documents it as unused.
Because the operator constructs `VertxHttpClientFactory` explicitly, users
cannot
switch to a client that sends pings without patching the operator.
## Suggested fixes
Any of these would resolve it:
1. Use (or allow configuring) the OkHttp HTTP client, which honours
`websocketPingInterval`.
2. Keep Vert.x but send periodic WebSocket pings on watch connections (or
get this
fixed in fabric8's Vert.x client, see below).
3. Make the HTTP client factory pluggable so operators on such platforms can
pick
one.
Separately, the liveness probe does not restart the operator while informers
stay
`UNHEALTHY`, so the operator can run for hours with a stale cache.
## Related
- fabric8io/kubernetes-client#7450: Vert.x `HttpClosedException` was not
retried
(closed 2026-03-20). With 7.8.0 the watches do reconnect, but the idle
drops
continue.
- A matching fabric8 issue for the missing ping support in `httpclient-vertx`
should be filed alongside this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
