[
https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187883#comment-14187883
]
Jing Zhao commented on HDFS-7276:
---------------------------------
Thanks for working on this, Nicholas! The patch looks pretty good to me. Here
are some of my thoughts:
# Do we want to have a mechanism to decrease the memory usage held by the
FixedLengthManager#freeQueue after the peak passes? Also maybe we still need a
way to clear FixedLengthManager from the ManagerMap for long running client?
# If the size of the packet header is fixed, maybe we can still use the
"powers of two" strategy here (exclude the header size when calculating the
size, and only count it in when do the real allocation). In this way we can
bound the max number of FixedLengthManager in ManagerMap.
# Can we directly allocate the byte array for heartbeat packet?
# Nit: It may be better to let DFSOutputStream#recycleBuffer not direct using
byteArrayManager (and call getByteArrayManager() instead).
# Nit: in the javadoc of ManagerMap, bye should be byte
# Maybe we can rename FixedLengthManager#arrayLength to byteArrayLength?
I think we can do #1 and #2 in separate jiras. For the current patch, maybe we
can have a configuration key to turn the feature off.
> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>
> Key: HDFS-7276
> URL: https://issues.apache.org/jira/browse/HDFS-7276
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Tsz Wo Nicholas Sze
> Attachments: h7276_20141021.patch, h7276_20141022.patch,
> h7276_20141023.patch, h7276_20141024.patch, h7276_20141027.patch,
> h7276_20141027b.patch, h7276_20141028.patch
>
>
> When there are a lot of DFSOutputStream's writing concurrently, the number of
> outstanding packets could be large. The byte arrays created by those packets
> could occupy a lot of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
