Meng-Shuan Tsai created HDDS-16373:
--------------------------------------
Summary: InterSCMGrpcClient treats the checkpoint download
deadline as seconds instead of milliseconds
Key: HDDS-16373
URL: https://issues.apache.org/jira/browse/HDDS-16373
Project: Apache Ozone
Issue Type: Bug
Reporter: Meng-Shuan Tsai
Assignee: Meng-Shuan Tsai
h2. Problem
{{InterSCMGrpcClient}} reads {{ozone.scm.ha.grpc.deadline.interval}} in
milliseconds, then passes the same numeric value to gRPC as seconds. The
default 30-minute deadline therefore becomes ~20 days (1000x).
This is a bug in SCM HA snapshot catch-up and Recon SCM DB sync. Successful
downloads that finish within 30 minutes are unaffected.
h2. Root cause
Config default is 30 minutes in both places:
* {{{}ozone-default.xml{}}}: {{ozone.scm.ha.grpc.deadline.interval=30m}}
* {{ScmConfigKeys.OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL_DEFAULT = 30 * 60 *
1000L}}
The client converts that to milliseconds correctly:
{code:java}
timeout = conf.getTimeDuration(
ScmConfigKeys.OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL,
ScmConfigKeys.OZONE_SCM_HA_GRPC_DEADLINE_INTERVAL_DEFAULT,
TimeUnit.MILLISECONDS); // 1,800,000 ms
{code}
Then applies it with the wrong unit:
{code:java}
client = InterSCMProtocolServiceGrpc.newStub(channel)
.withDeadlineAfter(timeout, TimeUnit.SECONDS); // 1,800,000 seconds ≈ 20
days
{code}
{{timeout}} is {{final}} and is not converted between the two calls.
Source:
*
[InterSCMGrpcClient.java|https://github.com/apache/ozone/blob/master/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcClient.java]
*
[ScmConfigKeys.java|https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java]
h2. Impact
Two production callers create this client and wait with
{{CompletableFuture.get()}} (no timeout):
# SCM follower install-snapshot: {{SCMStateMachine}} → {{SCMHAManagerImpl}} →
{{SCMSnapshotProvider}}
# Recon SCM DB sync: {{StorageContainerServiceProviderImpl}}
If the gRPC stream stay pending, the 30-minute deadline does not fire. Catch-up
/ sync is delayed until some other completion signal, an interrupt, or the
inflated gRPC deadline (~20 days).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
