[
https://issues.apache.org/jira/browse/HDDS-17310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng reassigned HDDS-17310:
---------------------------------
Assignee: Siyao Meng
Description:
When an OM or SCM host does not respond to TCP connection attempts at all (host
down behind a router, NIC or firewall dropping packets, stale Kubernetes pod
IP), the client spends about 15 minutes on that one server before failing over
to the next OM or SCM.
The OM and SCM failover proxy providers create RPC proxies with
{{RetryPolicies.failoverOnNetworkException(0)}} so that the IPC layer does not
retry on the same server and the provider fails over instead. That policy only
applies to connection failures such as connection refused. A
{{ConnectTimeoutException}} is retried separately in
{{Client.Connection#setupConnection}} up to
{{ipc.client.connect.max.retries.on.timeouts}} (Hadoop default 45) times, each
waiting {{ipc.client.connect.timeout}} (default 20s), which is about 15 minutes
per failover attempt. Ozone does not override either key. HDDS-16376 shows this
in practice ({{Retrying connect to server: scm1:9860. Already tried 44 time(s);
maxRetries=45}}).
HDFS avoids this for HA NameNode proxies by setting
{{ipc.client.connect.max.retries.on.timeouts}} to 0 by default
({{dfs.client.failover.connection.retries.on.timeouts}}).
Proposal: for RPC proxies created by the OM and SCM failover proxy providers,
use {{ipc.client.connect.max.retries.on.timeouts=0}} unless the key is set
explicitly. The count is read per connection ({{ConnectionId}}), so it applies
even though the IPC {{Client}} is cached (see HDDS-16382, which covers the
separate RPC timeout and connect timeout tuning).
was:placeholder
> OM and SCM clients retry a connect timeout on the same server for about 15
> minutes
> ----------------------------------------------------------------------------------
>
> Key: HDDS-17310
> URL: https://issues.apache.org/jira/browse/HDDS-17310
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client, SCM Client
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> When an OM or SCM host does not respond to TCP connection attempts at all
> (host down behind a router, NIC or firewall dropping packets, stale
> Kubernetes pod IP), the client spends about 15 minutes on that one server
> before failing over to the next OM or SCM.
> The OM and SCM failover proxy providers create RPC proxies with
> {{RetryPolicies.failoverOnNetworkException(0)}} so that the IPC layer does
> not retry on the same server and the provider fails over instead. That policy
> only applies to connection failures such as connection refused. A
> {{ConnectTimeoutException}} is retried separately in
> {{Client.Connection#setupConnection}} up to
> {{ipc.client.connect.max.retries.on.timeouts}} (Hadoop default 45) times,
> each waiting {{ipc.client.connect.timeout}} (default 20s), which is about 15
> minutes per failover attempt. Ozone does not override either key. HDDS-16376
> shows this in practice ({{Retrying connect to server: scm1:9860. Already
> tried 44 time(s); maxRetries=45}}).
> HDFS avoids this for HA NameNode proxies by setting
> {{ipc.client.connect.max.retries.on.timeouts}} to 0 by default
> ({{dfs.client.failover.connection.retries.on.timeouts}}).
> Proposal: for RPC proxies created by the OM and SCM failover proxy providers,
> use {{ipc.client.connect.max.retries.on.timeouts=0}} unless the key is set
> explicitly. The count is read per connection ({{ConnectionId}}), so it
> applies even though the IPC {{Client}} is cached (see HDDS-16382, which
> covers the separate RPC timeout and connect timeout tuning).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
