[
https://issues.apache.org/jira/browse/HDDS-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17322:
------------------------------
Epic Link: HDDS-17321
> Datanode heartbeat to an unreachable SCM or Recon can block about 15 minutes
> on connect timeout retries
> -------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17322
> URL: https://issues.apache.org/jira/browse/HDDS-17322
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> {{SCMConnectionManager}} creates the Datanode's RPC proxies to SCM and Recon
> with {{retryUpToMaximumCountWithFixedSleep(15, 1s)}}. That policy only covers
> connection failures such as "connection refused". A connect timeout (the host
> drops packets: a stopped VM, a firewall DROP rule, a stale Kubernetes pod IP)
> is handled separately by the forked {{ipc_.Client}}, which retries it
> {{ipc.client.connect.max.retries.on.timeouts}} (45) times with
> {{ipc.client.connect.timeout}} (20s) each. By calculation that is about 15
> minutes per call. Neither key is overridden for these proxies.
> {{RunningDatanodeState}} waits for an endpoint task for at most one heartbeat
> interval but does not cancel it, and each endpoint runs its tasks on a single
> thread with an unbounded queue (see HDDS-17269). So while an SCM is
> unreachable this way, one heartbeat task is queued per interval and each one
> blocks for about 15 minutes. Each missed heartbeat takes about 15 minutes
> instead of one interval, so re-resolving the SCM address after
> {{hdds.heartbeat.address.refresh.missed.count.threshold}} missed heartbeats
> (when enabled) is delayed by the same factor.
> Proposed fix: create these proxies with
> {{ipc.client.connect.max.retries.on.timeouts}} set to 0 unless it is set
> explicitly, as HDDS-17310 does for the OM and SCM client failover proxies,
> and let the Datanode's own retry policy and heartbeat cycle handle the
> retries.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
