[
https://issues.apache.org/jira/browse/HDDS-17310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17310:
------------------------------
Status: Patch Available (was: Open)
> OM and SCM clients retry a connect timeout on the same server for about 15
> minutes
> ----------------------------------------------------------------------------------
>
> Key: HDDS-17310
> URL: https://issues.apache.org/jira/browse/HDDS-17310
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client, SCM Client
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> When an OM or SCM host does not respond to TCP connection attempts at all
> (host down behind a router, NIC or firewall dropping packets, stale
> Kubernetes pod IP), the client spends about 15 minutes on that one server
> before failing over to the next OM or SCM.
> The OM and SCM failover proxy providers create RPC proxies with
> {{RetryPolicies.failoverOnNetworkException(0)}} so that the IPC layer does
> not retry on the same server and the provider fails over instead. That policy
> only applies to connection failures such as connection refused. A
> {{ConnectTimeoutException}} is retried separately in
> {{Client.Connection#setupConnection}} up to
> {{ipc.client.connect.max.retries.on.timeouts}} (Hadoop default 45) times,
> each waiting {{ipc.client.connect.timeout}} (default 20s), which is about 15
> minutes per failover attempt. Ozone does not override either key. HDDS-16376
> shows this in practice ({{Retrying connect to server: scm1:9860. Already
> tried 44 time(s); maxRetries=45}}).
> HDFS avoids this for HA NameNode proxies by setting
> {{ipc.client.connect.max.retries.on.timeouts}} to 0 by default
> ({{dfs.client.failover.connection.retries.on.timeouts}}).
> Proposal: for RPC proxies created by the OM and SCM failover proxy providers,
> use {{ipc.client.connect.max.retries.on.timeouts=0}} unless the key is set
> explicitly. The count is read per connection ({{ConnectionId}}), so it
> applies even though the IPC {{Client}} is cached (see HDDS-16382, which
> covers the separate RPC timeout and connect timeout tuning).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
