[
https://issues.apache.org/jira/browse/HDDS-11343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-11343:
------------------------------
Epic Link: HDDS-17321
> S3G randomly selects retry time when retrying OM
> ------------------------------------------------
>
> Key: HDDS-11343
> URL: https://issues.apache.org/jira/browse/HDDS-11343
> Project: Apache Ozone
> Issue Type: Improvement
> Components: s3gateway
> Affects Versions: 1.4.0
> Reporter: JiangHua Zhu
> Assignee: Chu Cheng Li
> Priority: Major
>
> When S3G retries OM, the rpc sleep time is the product of
> {ozone.client.wait.between.retries.millis} and
> {ozone.client.failover.max.attempts}.
> sleeping time =
> {ozone.client.wait.between.retries.millis}*{ozone.client.failover.max.attempts}
> When retry=1, sleep time = 2000ms.
> When retry=2, sleep time = 4000ms.
> ......
> Here are some S3Gateway logs:
> {code:java}
> 2024-08-19 10:04:01,978 [Thread-13636] INFO
> org.apache.hadoop.io.retry.RetryInvocationHandler:
> com.google.protobuf.ServiceException: java.net.ConnectException: Call From
> xxx.xxx.xxx.xxx/xx.xx.xx.xx to xxx.xxx.xxx.xxx:9862 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> $Proxy124.submitRequest over nodeId=null,nodeAddress=xxx.xxx.xxx.xxx:9862.
> Trying to failover after sleeping for 24000ms. Current retry count: 0.
> {code}
> {code:java}
> 2024-08-19 10:05:51,985 [Thread-14359] INFO
> org.apache.hadoop.io.retry.RetryInvocationHandler:
> com.google.protobuf.ServiceException: java.net.ConnectException: Call From
> xxx.xxx.xxx.xxx/xxx.xxx.xxx.xxx to xx.xx.xx.xx:9862 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> $Proxy124.submitRequest over nodeId=null,nodeAddress=xxx.xxx.xxx.xxx:9862.
> Trying to failover after sleeping for 134000ms. Current retry count: 0.
> {code}
> We see that the sleep time reaches tens of seconds or even longer. Each
> increase here is very expensive.
> We can optimize as much as possible without changing the linear probability.
> When choosing the sleep time, you can get a random value from the calculated
> value.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
