[
https://issues.apache.org/jira/browse/HDDS-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17318:
------------------------------
Status: Patch Available (was: Open)
> SCM client retries non-retriable server errors, safemode wait ignores its
> timeout
> ---------------------------------------------------------------------------------
>
> Key: HDDS-17318
> URL: https://issues.apache.org/jira/browse/HDDS-17318
> Project: Apache Ozone
> Issue Type: Improvement
> Components: SCM Client
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> *SCM client retries errors that will not change.*
> {{SCMHAUtils.getRetryAction}} fails over and retries any exception that is
> not in its known lists, including an ordinary error returned by SCM. With the
> defaults ({{hdds.scmclient.max.retry.timeout=10m}},
> {{hdds.scmclient.failover.retry.interval=2s}}) that is 300 attempts, about 10
> minutes. For example, {{ozone admin containerbalancer stop}} when the
> balancer is not running, or {{containerbalancer start}} with an invalid
> option, prints "Retrying" for 10 minutes before showing the error.
> SCM already converts the errors worth retrying (not leader, leader not ready,
> Ratis reconfiguration, and so on) to {{ServerNotLeaderException}},
> {{RetriableWithFailOverException}} or {{RetriableWithNoFailoverException}},
> and Ratis timeouts arrive as {{SCMException}}, which already fails without
> retry. So the client can fail immediately on any other exception returned by
> SCM. Connection failures and timeouts on the client side are still retried.
> *{{ozone admin safemode wait -t}} does not honor the timeout.* The deadline
> is only checked between calls to SCM, and each call can take the full SCM
> client retry budget. Its "SCM is not available (yet?)" branch only catches
> {{InterruptedException}}, so an error from SCM ends the command instead of
> retrying until the deadline.
> Proposed fix: fail fast on non-retriable exceptions returned by SCM, and run
> the {{inSafeMode}} call with the remaining time as its deadline, retrying on
> errors until the timeout.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
