Siyao Meng created HDDS-17322:
---------------------------------
Summary: Datanode heartbeat to an unreachable SCM or Recon can
block about 15 minutes on connect timeout retries
Key: HDDS-17322
URL: https://issues.apache.org/jira/browse/HDDS-17322
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Reporter: Siyao Meng
Assignee: Siyao Meng
{{SCMConnectionManager}} creates the Datanode's RPC proxies to SCM and Recon
with {{retryUpToMaximumCountWithFixedSleep(15, 1s)}}. That policy only covers
connection failures such as "connection refused". A connect timeout (the host
drops packets: a stopped VM, a firewall DROP rule, a stale Kubernetes pod IP)
is handled separately by the forked {{ipc_.Client}}, which retries it
{{ipc.client.connect.max.retries.on.timeouts}} (45) times with
{{ipc.client.connect.timeout}} (20s) each. By calculation that is about 15
minutes per call. Neither key is overridden for these proxies.
{{RunningDatanodeState}} waits for an endpoint task for at most one heartbeat
interval but does not cancel it, and each endpoint runs its tasks on a single
thread with an unbounded queue (see HDDS-17269). So while an SCM is unreachable
this way, one heartbeat task is queued per interval and each one blocks for
about 15 minutes. Each missed heartbeat takes about 15 minutes instead of one
interval, so re-resolving the SCM address after
{{hdds.heartbeat.address.refresh.missed.count.threshold}} missed heartbeats
(when enabled) is delayed by the same factor.
Proposed fix: create these proxies with
{{ipc.client.connect.max.retries.on.timeouts}} set to 0 unless it is set
explicitly, as HDDS-17310 does for the OM and SCM client failover proxies, and
let the Datanode's own retry policy and heartbeat cycle handle the retries.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
