[ https://issues.apache.org/jira/browse/HDFS-17364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824416#comment-17824416 ]
ASF GitHub Bot commented on HDFS-17364: --------------------------------------- bbeaudreault commented on code in PR #6514: URL: https://github.com/apache/hadoop/pull/6514#discussion_r1516181907 ########## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java: ########## @@ -3159,6 +3165,21 @@ private void initThreadsNumForStripedReads(int numThreads) { } } + private void initBufferPoolForStripedReads(boolean useWeakReference) { + if (STRIPED_READ_BUFFER_POOL != null) { + return; + } + synchronized (DFSClient.class) { Review Comment: @Hexiaoqiao thanks for review. For this block, it's sort of modeled after other examples in DFSClient, such [as initializing the striped read thread pool](https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L3150). I think the idea is that DFSClient could easily be used in multiple threads, so we want to avoid double initializing the shared resource. > Use WeakReferencedElasticByteBufferPool in DFSStripedInputStream > ---------------------------------------------------------------- > > Key: HDFS-17364 > URL: https://issues.apache.org/jira/browse/HDFS-17364 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Bryan Beaudreault > Priority: Major > Labels: pull-request-available > > DFSStripedInputStream uses ElasticByteBufferPool to allocate byte buffers for > the "curStripeBuf". This is used for non-positional (stateful) reads and is > allocated with a size of numDataBlocks * cellSize. For RS-6-3-1024k, that > means each DFSStripedInputStream could allocate a 6mb buffer. When the IS is > finished, the buffer is put back in the pool. Over time and with spikes of > concurrent reads, the pool grows and most of the buffers sit there unused. > > WeakReferencedElasticByteBufferPool was introduced HADOOP-18105 and mitigates > this issue because the excess buffers can be GC'd once they are no longer > needed. We should use this same pool in DFSStripedInputStream -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org