matanper commented on issue #69984: URL: https://github.com/apache/doris/issues/69984#issuecomment-6578317645
Validated the proposed fix under live Stream Load traffic on a Kubernetes cluster (`enable_fqdn_mode=true`, doris-operator), restarting one BE so it came back on a new pod IP: | | Before the fix | With the fix | |---|---|---| | `Host is down` window on the other BEs after the restarted BE was Ready | ~55 s (until the periodic `DNSCache` refresh) | ~20 ms | With the fix, the first failed tablet-writer open to the old IP triggers an immediate re-resolve. `update hostname …'s ip to <new ip>` is logged in the same millisecond as that first failure, and failures stop about 20 ms later. The change includes the `DNSCache::refresh()` / `BrpcClientCache::get_client()` re-resolve described above, made safe for concurrent callers (single-flight per hostname, compare-and-set cache updates), plus new unit tests. I'll open the PR against master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
