Wei-Chiu Chuang created HDDS-16417:
--------------------------------------
Summary: InterSCMGrpcClient mis-applies
ozone.scm.ha.grpc.deadline.interval (MILLISECONDS vs SECONDS)
Key: HDDS-16417
URL: https://issues.apache.org/jira/browse/HDDS-16417
Project: Apache Ozone
Issue Type: Bug
Components: SCM HA
Reporter: Wei-Chiu Chuang
h3. Problem
{{ozone.scm.ha.grpc.deadline.interval}} is documented as the deadline for SCM
DB checkpoint download over Inter-SCM gRPC (default {{30m}} in
{{ozone-default.xml}}).
{{InterSCMGrpcClient}} reads the config with {{getTimeDuration(...,
TimeUnit.MILLISECONDS)}}, so {{30m}} becomes {{1_800_000}} (milliseconds).
The gRPC stub deadline is then set with:
{code}
withDeadlineAfter(timeout, TimeUnit.SECONDS);
{code}
So the numeric value is interpreted as *seconds*, not milliseconds. With the
default config this is ~1,800,000 seconds (~21 days), not 30 minutes. The
documented deadline is not enforced.
h3. Affected code
*
{{hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcClient.java}}
* Config: {{ScmConfigKeys.OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL}} /
{{ozone-default.xml}}
Used when a follower (or Recon) downloads the leader SCM RocksDB checkpoint
during SCM HA install snapshot ({{SCMSnapshotProvider.getSCMDBSnapshot}}).
h3. Expected behavior
A {{30m}} setting should expire the client RPC after 30 minutes of wall-clock
time (gRPC {{DEADLINE_EXCEEDED}}), allowing operators to tune large {{scm.db}}
transfers.
h3. Suggested fix
Use {{TimeUnit.MILLISECONDS}} in {{withDeadlineAfter}}, or read the duration
with {{TimeUnit.SECONDS}} and pass seconds consistently. Add a unit test that a
small value (e.g. {{1s}}) fails a slow/hung download within expected bounds.
h3. Impact
Operators may believe snapshot downloads are capped at 30m while the effective
gRPC deadline is orders of magnitude larger; troubleshooting snapshot loops may
incorrectly rule out client deadline expiry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
