[
https://issues.apache.org/jira/browse/HDFS-17676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17901827#comment-17901827
]
Longping Jie commented on HDFS-17676:
-------------------------------------
I'd be happy to submit this code if necessary.
> The frequency of submitting DFSInputStream refresh block locations in the
> LocatedBlocksRefresher thread is too fast.
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17676
> URL: https://issues.apache.org/jira/browse/HDFS-17676
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: dfsclient
> Affects Versions: 3.3.6
> Reporter: Longping Jie
> Priority: Major
> Attachments: image-2024-11-29-10-12-45-483.png
>
>
> LocatedBlocksRefresher thread is responsible for submitting the task of
> DFSInputStream refresh block locations.
> When a region server opens tens of thousands of DFSInputstreams, these
> DFSInputstream refresh block locations will be executed in a short period of
> time, which will put great pressure on the datanode, create a large number of
> DataXceiver threads in a short period of time, and cause GC fluctuations,
> resulting in time-consuming hbase queries. Burr appears
> !image-2024-11-29-10-12-45-483.png!
> Even if you set the size of refreshThreadPool to 1, you can't limit the rate
> well, consider adding a rate limiting mechanism, which can be enabled by
> parameter configuration to limit the refresh frequency.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
