dongjoon-hyun commented on issue #862: URL: https://github.com/apache/spark-kubernetes-operator/issues/862#issuecomment-5790752314
Thank you for the detailed report and measurements, @gada121982. I verified the root cause on our side: - `KubernetesClientFactory` explicitly uses `VertxHttpClientFactory` (since SPARK-53647), and there is no way to choose a different HTTP client. - `kubernetes-httpclient-vertx` 7.8.0 has no reference to `websocketPingInterval`, and neither does fabric8 `main`. It only configures an idle timeout. So idle watch connections carry no traffic, and a load balancer with a 50s idle cutoff closes them. A few comments: 1. **Missed events vs. delayed events.** A reconnect resumes from the last `resourceVersion`, so events should be replayed unless the server returns `410 Gone`. `received 335 times without progress` suggests the main effect is the growing reconnect backoff, which delays events past the timeouts, rather than lost events. Could you share the full stack trace of the `Timed out waiting for [60000] milliseconds for [Pod]` exception? That would show which code path waits on the pod. 2. **Liveness probe.** `/healthz` does check the informer health. However, with the default `periodSeconds: 10` and `failureThreshold: 3`, a short `UNHEALTHY` window during each reconnect does not trigger a restart. A restart would not help here anyway, because the new watches would be dropped the same way. 3. **Fix direction.** Since Apache Spark uses the same Vert.x HTTP client, the right place to fix this is fabric8's `httpclient-vertx`. It should send periodic WebSocket pings based on `Config#websocketPingInterval`, like `httpclient-okhttp` does. We would prefer that over switching this project to OkHttp or making the HTTP client factory pluggable. Please feel free to file the fabric8 issue and link it here. 4. **Workaround for now.** If your provider supports it, increase the idle timeout of the API server load balancer above the bookmark interval (for example, a few minutes). Also note that the `main` branch now requires K8s v1.35 or newer. Please check the reproduction against the latest release as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
