Wei-Chiu Chuang created HDDS-16418:
--------------------------------------
Summary: Make Inter-SCM gRPC timeouts and keepalive
operator-tunable
Key: HDDS-16418
URL: https://issues.apache.org/jira/browse/HDDS-16418
Project: Apache Ozone
Issue Type: Improvement
Components: SCM HA
Reporter: Wei-Chiu Chuang
h3. Problem
SCM HA followers (and Recon) download the leader SCM RocksDB checkpoint over
Inter-SCM gRPC during install snapshot. Transfers can run for a long time and
depend on network stability, {{scm.db}} size, and disk throughput.
Today operators have almost no way to tune gRPC timeout and connection behavior
for this path:
* *Client* ({{InterSCMGrpcClient}}): only
{{ozone.scm.ha.grpc.deadline.interval}} exists for the stub RPC deadline. No
documented client keepalive / idle settings on the channel. (See also
HDDS-16417 for incorrect application of the deadline value.)
* *Server* ({{InterSCMGrpcProtocolService}}): only port
({{ozone.scm.grpc.port}}) and {{maxInboundMessageSize}} are set. Keepalive, max
connection idle, and related Netty server options use gRPC-java defaults and
are not exposed in {{ozone-default.xml}}.
By contrast, datanode container gRPC in {{XceiverServerGrpc}} sets explicit
{{maxConnectionIdle}}, {{keepAliveTime}}, and {{keepAliveTimeout}} (currently
hardcoded, not config either, but documented in code comments).
Operators troubleshooting snapshot download stalls, proxy idle timeouts, or
load-balancer disconnects cannot align Inter-SCM gRPC behavior with their
environment without code changes.
h3. Proposal
Introduce *operator-tunable* Inter-SCM gRPC timeout and keepalive settings
under a consistent {{ozone.scm.ha.grpc.*}} prefix (names to be finalized in
review), for example:
* Client RPC deadline (retain / clarify {{ozone.scm.ha.grpc.deadline.interval}})
* Server: {{maxConnectionIdle}}, {{keepAliveTime}}, {{keepAliveTimeout}} (and
any other knobs needed for long-lived streaming RPCs)
* Client channel: matching keepalive / idle settings where applicable so client
and server policies are coherent
Defaults should remain safe for large checkpoint downloads (generous
idle/deadline vs datanode chunk RPCs). Document each property in
{{ozone-default.xml}} with units and interaction with TLS.
h3. Affected code
*
{{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcClient.java}}
*
{{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcProtocolService.java}}
*
{{hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java}}
* {{hadoop-ozone/recon/.../StorageContainerServiceProviderImpl.java}} (Recon
uses {{InterSCMGrpcClient}})
h3. Acceptance criteria
* New config keys wired on both Inter-SCM gRPC client and server builders.
* Defaults preserve current effective behavior (after fixing HDDS-16417
deadline units).
* Unit or integration test coverage that config values are read and applied
(mock or small local server/client test acceptable).
* Release note if defaults or operator-visible behavior changes.
h3. Related
* HDDS-16417 — deadline interval TimeUnit bug on the existing client property.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
