[
https://issues.apache.org/jira/browse/HDDS-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17316:
------------------------------
Description:
At startup, OM and Datanode load the current secret key from SCM in
{{DefaultSecretKeySignerClient.loadInitialSecretKey}}. While SCM has not
initialized secret keys yet ({{SECRET_KEY_NOT_INITIALIZED}}), it makes up to
100 attempts with exponential backoff that restarts every 10 retries. The base
wait goes 4s, 8s, ..., 1024s, then 2s, randomized by +/-50%, so a single wait
can be up to about 25 minutes and the total about 5.7 hours. The code comment
("1 2 4 8 ... 512", "around 200min") understates both.
So once SCM finishes initializing its keys, an OM or Datanode can stay stuck
for many more minutes before trying again.
Proposed fix: keep the exponential backoff but cap the base wait at 16s (4s,
8s, 16s, 16s, ...), so a single wait is at most 23s and the total for 100
attempts is around 26 minutes.
was:
At startup, OM and Datanode load the current secret key from SCM in
{{DefaultSecretKeySignerClient.loadInitialSecretKey}}. While SCM has not
initialized secret keys yet ({{SECRET_KEY_NOT_INITIALIZED}}), it retries up to
100 times with exponential backoff that restarts every 10 retries. The base
wait goes 2s, 4s, ..., 1024s, randomized by +/-50%, so a single wait can be up
to about 25 minutes and the total about 5.7 hours. The code comment ("1 2 4 8
... 512", "around 200min") understates both.
So once SCM finishes initializing its keys, an OM or Datanode can stay stuck
for many more minutes before trying again.
Proposed fix: keep the exponential backoff but cap the base wait at 16s (2s,
4s, 8s, 16s, 16s, ...), so a single wait is at most 24s and the total for 100
retries is around 26 minutes.
> Cap the wait between retries when loading the initial secret key from SCM
> -------------------------------------------------------------------------
>
> Key: HDDS-17316
> URL: https://issues.apache.org/jira/browse/HDDS-17316
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Security
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> At startup, OM and Datanode load the current secret key from SCM in
> {{DefaultSecretKeySignerClient.loadInitialSecretKey}}. While SCM has not
> initialized secret keys yet ({{SECRET_KEY_NOT_INITIALIZED}}), it makes up to
> 100 attempts with exponential backoff that restarts every 10 retries. The
> base wait goes 4s, 8s, ..., 1024s, then 2s, randomized by +/-50%, so a single
> wait can be up to about 25 minutes and the total about 5.7 hours. The code
> comment ("1 2 4 8 ... 512", "around 200min") understates both.
> So once SCM finishes initializing its keys, an OM or Datanode can stay stuck
> for many more minutes before trying again.
> Proposed fix: keep the exponential backoff but cap the base wait at 16s (4s,
> 8s, 16s, 16s, ...), so a single wait is at most 23s and the total for 100
> attempts is around 26 minutes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
