matanper opened a new issue, #70382: URL: https://github.com/apache/doris/issues/70382
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, FQDN mode, BEs as Kubernetes pods). master is the same: - `fe/fe-core/src/main/java/org/apache/doris/common/DNSCache.java:48`: refresh scheduled every 60 s. - `DNSCache.java:60`: `get` is `computeIfAbsent`. - `fe/fe-core/src/main/java/org/apache/doris/rpc/BackendServiceProxy.java:121-122` / `:149`: `getProxy` dials the cached IP and rebuilds a non-normal channel from the same IP. ### Description Since #59904, FE→BE brpc calls dial the IP returned by the FE `DNSCache` instead of the hostname. `DNSCache.get` returns the cached entry (`computeIfAbsent`) and only the periodic task re-resolves, every 60 s (`DNSCache.java:48`). When a BE pod restarts and comes back with a new pod IP under the same hostname, the FE keeps dialing the old IP until the next periodic refresh. `getProxy` does detect the broken channel (`!isNormalState()`, `BackendServiceProxy.java:149`) and rebuilds it, but from the same stale cached IP. As a result, every query that schedules a fragment on that BE fails with `send fragments failed ... UNAVAILABLE: io exception` for up to ~60 s after the BE has rejoined and is reported alive. In a rolling restart of the BEs this happens on every BE rejoin. #69450 fixed the same stale-IP problem for heartbeat connections only. Query RPCs still use the stale entry. ### Solution Re-resolve on demand when there is evidence the IP may have changed: 1. Add `DNSCache.invalidate(hostname)`. The next `get` does a forced lookup, serialized per host. A slower periodic refresh must not overwrite a newer result, and the last known IP is kept as a fallback if DNS fails. 2. Drop the cached `BackendServiceClient` for that address so the next call dials the new IP. 3. Trigger both when: - a BE RPC fails with `UNAVAILABLE` / connect failure; - `getProxy` finds a non-normal channel; - the heartbeat reports a changed BE start time (restart). The open PR #69423 contains a version of 1–3 as part of a much larger change (graceful rolling restarts, Stream Load retries, BE changes and a new global variable). Its latest review notes that an invalidation can be lost during a hostname's first lookup. This issue asks for the narrow default-mode fix independently, so it can land and be backported to 4.1 on its own. ### How to Reproduce? 1. Run a cluster in FQDN mode where a BE's hostname maps to a new IP after a restart, for example a Kubernetes BE pod deleted and recreated. 2. Run queries that use that BE continuously. 3. Restart the BE. Once it is alive again, queries scheduled on it keep failing with `UNAVAILABLE: io exception` until the FE log shows `IP for hostname ... has changed from <old> to <new>` (up to 60 s later). ### Anything Else? We have a patch that ports the FE DNS part of #69423 (with the first-lookup race handled) and can open a PR. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
