[ https://issues.apache.org/jira/browse/HDFS-13274?focusedWorklogId=787727&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-787727 ]
ASF GitHub Bot logged work on HDFS-13274: ----------------------------------------- Author: ASF GitHub Bot Created on: 05/Jul/22 04:20 Start Date: 05/Jul/22 04:20 Worklog Time Spent: 10m Work Description: ZanderXu opened a new pull request, #4531: URL: https://github.com/apache/hadoop/pull/4531 ### Description of PR Jira: [HDFS-13274](https://issues.apache.org/jira/browse/HDFS-13274) [HADOOP-13144](https://issues.apache.org/jira/browse/HADOOP-13144) introduces the ability to create multiple connections for the same user and use different sockets. The RouterRpcClient should use this approach to get a better throughput. In my practice, I configured `dfs.federation.router.connection.pool-size=6` and found some abnormal cases: - RBF frequently to create socket with NN during forwarding request. - There are a large number of connection threads with name node. (There are 70+ downstream name services) After tracing the code and found that we can optimize the mechanism of round-robin obtaining connection. - One `ConnectionContext` can support handler some concurrent requests at the same time, such as 5 - We can mark one `ConnectionContext` usable when the `numThreads` is less than `maxConcurrencyPerConn` - About `getConnection`, we can returns the previous usable connection in order - If there no usable connections, we can return one connection round-robin After using the above solution, we effectively controlled the number of connections between RBF and NameNode, and there is almost no the phenomenon of creating sockets during forwarding requests. Issue Time Tracking ------------------- Worklog Id: (was: 787727) Remaining Estimate: 0h Time Spent: 10m > RBF: Extend RouterRpcClient to use multiple sockets > --------------------------------------------------- > > Key: HDFS-13274 > URL: https://issues.apache.org/jira/browse/HDFS-13274 > Project: Hadoop HDFS > Issue Type: Sub-task > Reporter: Íñigo Goiri > Assignee: Íñigo Goiri > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > HADOOP-13144 introduces the ability to create multiple connections for the > same user and use different sockets. The RouterRpcClient should use this > approach to get a better throughput. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org