[
https://issues.apache.org/jira/browse/HDFS-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17791897#comment-17791897
]
ASF GitHub Bot commented on HDFS-17242:
---------------------------------------
ayushtkn commented on code in PR #6227:
URL: https://github.com/apache/hadoop/pull/6227#discussion_r1411597652
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##########
@@ -6525,6 +6525,21 @@
If the namespace is DEFAULT, it's best to change this conf to other
value.
</description>
</property>
+ <property>
+ <name>dfs.client.congestion.backoff.mean.time</name>
+ <value>5000</value>
+ <description>
+ The mean milliseconds which is used to compute client congestion backoff
sleep time.
Review Comment:
The mean time in milliseconds which is used to compute client congestion
backoff sleep time.
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:
##########
@@ -564,6 +564,34 @@ private DataStreamer(HdfsFileStatus stat, ExtendedBlock
block,
this.addBlockFlags = flags;
this.maxPipelineRecoveryRetries = conf.getMaxPipelineRecoveryRetries();
this.markSlowNodeAsBadNodeThreshold =
conf.getMarkSlowNodeAsBadNodeThreshold();
+ congestionBackOffMeanTimeInMs = dfsClient.getConfiguration().getInt(
+ HdfsClientConfigKeys.DFS_CLIENT_CONGESTION_BACKOFF_MEAN_TIME,
+ HdfsClientConfigKeys.DFS_CLIENT_CONGESTION_BACKOFF_MEAN_TIME_DEFAULT);
+ congestionBackOffMaxTimeInMs = dfsClient.getConfiguration().getInt(
+ HdfsClientConfigKeys.DFS_CLIENT_CONGESTION_BACKOFF_MAX_TIME,
+ HdfsClientConfigKeys.DFS_CLIENT_CONGESTION_BACKOFF_MAX_TIME_DEFAULT);
+ if (congestionBackOffMeanTimeInMs <= 0 || congestionBackOffMaxTimeInMs <=
0 ||
+ congestionBackOffMaxTimeInMs < congestionBackOffMeanTimeInMs) {
Review Comment:
I don't think we need to have a redundant wrapping if condition, we can
directly go to if statements & change to defaults post checking the condition
in the same if block
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##########
@@ -6525,6 +6525,21 @@
If the namespace is DEFAULT, it's best to change this conf to other
value.
</description>
</property>
+ <property>
+ <name>dfs.client.congestion.backoff.mean.time</name>
+ <value>5000</value>
+ <description>
+ The mean milliseconds which is used to compute client congestion backoff
sleep time.
+ </description>
+ </property>
+ <property>
+ <name>dfs.client.congestion.backoff.max.time</name>
+ <value>50000</value>
+ <description>
+ The max milliseconds which is used to
Review Comment:
The max time in milliseconds which is used to restrict the upper limit
backoff sleep time for client.
> Make congestion backoff time configurable
> -----------------------------------------
>
> Key: HDFS-17242
> URL: https://issues.apache.org/jira/browse/HDFS-17242
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 3.4.0
> Reporter: farmmamba
> Assignee: farmmamba
> Priority: Minor
> Labels: pull-request-available
>
> Currently，if we enable congestion backoff, we will actually invoke
> backOffIfNecessary method. and the backoff time is computed using
> CONGESTION_BACKOFF_MEAN_TIME_IN_MS and CONGESTION_BACK_OFF_MAX_TIME_IN_MS
> which are hard-code. We should better make them configurable to compute
> backoff sleep time flexibly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
