[
https://issues.apache.org/jira/browse/HDFS-17685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17904826#comment-17904826
]
ASF GitHub Bot commented on HDFS-17685:
---------------------------------------
zeekling commented on code in PR #7215:
URL: https://github.com/apache/hadoop/pull/7215#discussion_r1880403316
##########
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/LeaseRenewer.java:
##########
@@ -378,17 +372,42 @@ public synchronized void closeClient(final DFSClient
dfsc) {
}
}
- //update renewal time
- if (renewal == dfsc.getConf().getHdfsTimeout()/2) {
- long min = HdfsConstants.LEASE_SOFTLIMIT_PERIOD;
- for(DFSClient c : dfsclients) {
- final int timeout = c.getConf().getHdfsTimeout();
- if (timeout > 0 && timeout < min) {
- min = timeout;
+ renewal = getNewRenewalIntervalMs(dfsclients);
+ }
+
+ @VisibleForTesting
+ static long getNewRenewalIntervalMs(Collection<DFSClient> dfsClients) {
+ // Update renewal time to the first applicable of:
+ // 1. Requested renewal time amongst all DFSClients, if requested and
smaller than the default
+ // renewal interval
+ // 2. Half the HDFS timeout amongst all DFSClients, if requested and
smaller than the default
+ // renewal interval
+ // 3. Default renewal time of HdfsConstants.LEASE_SOFTLIMIT_PERIOD / 2
+ // #2 exists because users with small timeouts want to find out quickly
when a NameNode dies,
+ // and a small lease renewal interval will help to inform them quickly.
See HDFS-278.
+ long min = HdfsConstants.LEASE_SOFTLIMIT_PERIOD / 2;
+ boolean leaseOverrideSet = false;
+ for (DFSClient c : dfsClients) {
+ final int newLeaseRenewalInterval =
c.getConf().getLeaseRenewalIntervalMs();
+ if (newLeaseRenewalInterval > 0 && newLeaseRenewalInterval <= min) {
+ min = newLeaseRenewalInterval;
+ leaseOverrideSet = true;
+ }
+ }
+ if (leaseOverrideSet) {
+ return min;
+ }
+
+ for (DFSClient c : dfsClients) {
Review Comment:
Can you combine the two loop to one?
> Option to explicitly choose DFS client lease renewal interval
> -------------------------------------------------------------
>
> Key: HDFS-17685
> URL: https://issues.apache.org/jira/browse/HDFS-17685
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient
> Reporter: Charles Connell
> Priority: Minor
> Labels: pull-request-available
>
> Currently, DFSClients send lease renewals to the NameNode at an interval
> equal to half of {{ipc.client.rpc-timeout.ms}}. This logic dates back to 2009
> in HDFS-278. At my company, we are interested in using short DFS client
> timeouts (< 10 seconds). However, we're currently hesitant to do so because
> that would flood the NameNode with lease renewals.
> I propose a setting {{dfs.client.lease.renewal.interval.ms}} that, if
> nonzero, would be used as the lease renewal interval instead of
> {{ipc.client.rpc-timeout.ms / 2}}. This would be useful for advanced users
> that want to control their RPC timeout and lease renewals separately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
