[
https://issues.apache.org/jira/browse/FLINK-41441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41441:
-----------------------------------
Labels: pull-request-available (was: )
> Operator silently runs on the JDK HTTP client instead of OkHttp
> ---------------------------------------------------------------
>
> Key: FLINK-41441
> URL: https://issues.apache.org/jira/browse/FLINK-41441
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
>
> h1. Problem
> Every official image since 1.13.0 logs this at startup:
> {noformat}
> 2026-11-10 11:44:05,954 i.f.k.c.u.HttpClientUtils [WARN ] The following
> httpclient factories were detected on your classpath:
> [io.fabric8.kubernetes.client.jdkhttp.JdkHttpClientFactory,
> io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory], multiple of which
> had the same priority (0) so one was chosen randomly. You should exclude
> dependencies that aren’t needed or use an explicit association of the
> HttpClient.Factory.
> {noformat}
> and then runs on fabric8's JDK HTTP client, although the project selects,
> documents and tests OkHttp ({{{}fabric8.httpclient.impl{}}} defaults to
> {{{}okhttp{}}}, the {{depend-on-okhttp4}} profile, the [IPv6
> Configuration|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/deployment/configuration/#ipv6-configuration]
> note in the configuration docs). The webhook builds its client from the same
> classpath, so it runs on the JDK client too.
> h1. Root cause
> * The shade plugin merges the two
> {{META-INF/services/io.fabric8.kubernetes.client.http.HttpClient$Factory}}
> files, JDK first. Neither factory overrides {{{}priority(){}}}, so fabric8
> takes the first one listed. "Randomly" in the warning is deterministic in
> practice: always the JDK factory.
> * JOSDK's {{operator-framework}} depends on {{kubernetes-httpclient-<impl>}}
> with the implementation taken from its parent pom property
> {{{}fabric8-httpclient-impl.name{}}}, which is {{jdk}} in every JOSDK 5.x
> release.
> *
> The poms only ever exclude the OkHttp client that fabric8 and JOSDK used to
> bring, and add the operator's own {{kubernetes-httpclient-<impl>}} next to
> it.
> Nothing excludes {{{}kubernetes-httpclient-jdk{}}}, so since the JOSDK 5.1
> upgrade the exclusions match nothing relevant and the JDK client rides in
> alongside the OkHttp one.
> h1. Impact
> The 2 HTTP client implementations differ in how they keep watches alive:
> * OkHttp pings every watch WebSocket every 30 seconds
> ({{{}kubernetes.websocket.ping.interval{}}}), so a connection that dies
> without a FIN or RST signal (API server replica replaced, load balancer state
> dropped) is detected within a minute.
> * The JDK implementation sends no pings and has no keepalive, so the same
> connection is only noticed when the client happens to write on it and the
> network answers with an RST.
> This behaviour was measured locally on Minikube with a connection made
> half-open on purpose:
> * operator built with a single JDK client factory: a FlinkDeployment created
> right after the failure stays without status, event or log line for about 2
> minutes, the health endpoint reports 200 throughout, then the watches are
> re-opened one by one over the next minutes;
> * operator built with a single OkHttp client factory: all connections
> dropped 46 seconds after the failure, re-established within 15 seconds of the
> network returning, the FlinkDeployment reconciled 30 seconds later, no silent
> window.
> So the bug does not make the operator permanently blind, it removes the
> fastest detection path and leaves a silent window of minutes after every
> connection loss, during which the health probe reports healthy.
> h1. Expected Behaviour
> * Exactly one fabric8 HTTP client factory on the classpath, the one selected
> by {{fabric8.httpclient.impl.}}
> * The implementation in use visible in the startup log;
> * A guard so that the next change of JOSDK's transitive client cannot
> regress this silently.
> h1. Related
> The production incident that led here (a FlinkDeployment ignored for 12+
> hours with a green health probe, fixed by an operator restart) had no
> connection failure in the operator log at all. Its watch was alive and
> silent, which points to a stale watch cache on the API server replica the
> operator was pinned to. Currently, no HTTP client can detect this issue, and
> this gap must be addressed in a separate ticket that improves health-check
> functionality.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
