Bryan Beaudreault created HDFS-17364:
----------------------------------------
Summary: Use WeakReferencedElasticByteBufferPool in
DFSStripedInputStream
Key: HDFS-17364
URL: https://issues.apache.org/jira/browse/HDFS-17364
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Bryan Beaudreault
DFSStripedInputStream uses ElasticByteBufferPool to allocate byte buffers for
the "curStripeBuf". This is used for non-positional (stateful) reads and is
allocated with a size of numDataBlocks * cellSize. For RS-6-3-1024k, that means
each DFSStripedInputStream could allocate a 6mb buffer. When the IS is
finished, the buffer is put back in the pool. Over time and with spikes of
concurrent reads, the pool grows and most of the buffers sit there unused.
WeakReferencedElasticByteBufferPool was introduced HADOOP-18105 and mitigates
this issue because the excess buffers can be GC'd once they are no longer
needed. We should use this same pool in DFSStripedInputStream
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
