Siyao Meng created HDDS-17325:
---------------------------------
Summary: GrpcOmTransport has no call deadline and ignores
interrupts, so a stuck OM blocks the client indefinitely
Key: HDDS-17325
URL: https://issues.apache.org/jira/browse/HDDS-17325
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Client
Reporter: Siyao Meng
Assignee: Siyao Meng
When {{ozone.om.transport.class}} selects the gRPC transport (not the default
since HDDS-9446, but used for S3 Gateway in some deployments),
{{GrpcOmTransport}}:
* calls {{submitRequest}} on a blocking stub without a deadline
({{withDeadlineAfter}}), and there is no config key for a call timeout. If the
OM accepts the request but never answers (e.g. a stuck handler), the call
blocks with no limit, and the failover and retry logic is never reached. A
half-open connection is only closed by TCP, after 15 minutes or more.
* sleeps between retries inside {{catch (Exception e)}}, which only logs, so an
interrupt is lost and the retry continues.
For S3 Gateway, each request that reaches the stuck OM holds a gateway thread
until the gateway is restarted.
Proposed fix: set a per-call deadline from a new config key (a
{{StatusRuntimeException}} already goes through the retry and failover policy),
and restore the interrupt flag and stop retrying when the thread is interrupted.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
