[
https://issues.apache.org/jira/browse/HDDS-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131306#comment-18131306
]
Ivan Andika commented on HDDS-17321:
------------------------------------
Thanks [~smeng] for raising this, +1 on this. We should bound all the timeout
to reasonable values.
I was genuinely surprised (and disheartened) when I found out that all the
default timeout can cause widespread outage in HDDS-16376 and has not been
revisited before.
> Bound client retry and timeout waits to avoid long silent hangs
> ---------------------------------------------------------------
>
> Key: HDDS-17321
> URL: https://issues.apache.org/jira/browse/HDDS-17321
> Project: Apache Ozone
> Issue Type: Epic
> Components: Ozone Client
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> Several Ozone clients wait far longer than needed, often with nothing
> printed, because a retry wait grows without bound, a timeout is retried many
> times on the same server, an error that will not change is retried, or a
> timeout setting is read in the wrong unit. For example, a CLI command started
> before a single OM is ready keeps sleeping long after OM is up, and an
> unreachable OM or SCM costs about 15 minutes per connection attempt.
> This epic groups fixes that keep these waits bounded and in line with the
> configured values, without changing behavior on the normal path.
> h3. Fixes
> * HDDS-17302 OM client retry wait time grows without bound with a single OM
> * HDDS-17310 OM and SCM clients retry a connect timeout on the same server
> for about 15 minutes
> * HDDS-17312 gRPC client read and EC write timeouts are truncated to whole
> seconds
> * HDDS-17314 Make hdds.heartbeat.recon.interval control the Datanode to Recon
> heartbeat interval
> * HDDS-17316 Cap the wait between retries when loading the initial secret key
> from SCM
> * HDDS-17318 SCM client retries non-retriable server errors, safemode wait
> ignores its timeout
> * HDDS-17320 SCM client does not recognize RetriableWithNoFailoverException
> returned by SCM
> h3. Open
> * HDDS-17322 Datanode heartbeat to an unreachable SCM or Recon can block
> about 15 minutes on connect timeout retries
> * HDDS-17323 OM admin client nests two retry loops and retries
> AccessControlException
> * HDDS-17324 Ozone CLI waits forever for the CA list when SCM has fewer CA
> certificates than configured SCM nodes
> * HDDS-17325 GrpcOmTransport has no call deadline and ignores interrupts, so
> a stuck OM blocks the client indefinitely
> * HDDS-17326 Client keeps waiting for ALL_COMMITTED on a pipeline with a
> failed follower
> * HDDS-17327 Topology-aware read tries an unresponsive nearest datanode first
> for every ReadChunk
> * HDDS-17328 EC read waits about 2 minutes on an unresponsive datanode before
> switching to reconstruction
> * HDDS-17329 Ratis client write can take about 2 minutes although
> hdds.ratis.client.request.write.timeout is 70s
> * HDDS-11343 (existing) OM "leader not ready" wait is not capped, and its
> counter is shared by all threads using one client
> Related: HDDS-16382, HDDS-4409
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
