[
https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186208#comment-14186208
]
Tsz Wo Nicholas Sze commented on HDFS-7276:
-------------------------------------------
> I think this might make sense to put into ClientContext.java. ...
The manager is going to be shared by all DFSOutputStreams for any client and
any user. How could it be put into ClientContext?
> ... I'm a bit concerned that if we have a lot of different sized packets, we
> may keep around a lot of extra stuff. ...
That's why we have countThreshold. For non-full and non-heartbeat packets, the
count should be always below the threshold.
> ... could we simply have ByteArrayManager return an array equal or bigger
> than the requested size? ...
I originally round up array sizes to powers of two. However, it is unfortunate
that our full package size is 64k + hearder length, which will round up to 128k.
> Limit the number of byte arrays used by DFSOutputStream
> -------------------------------------------------------
>
> Key: HDFS-7276
> URL: https://issues.apache.org/jira/browse/HDFS-7276
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Tsz Wo Nicholas Sze
> Attachments: h7276_20141021.patch, h7276_20141022.patch,
> h7276_20141023.patch, h7276_20141024.patch, h7276_20141027.patch,
> h7276_20141027b.patch
>
>
> When there are a lot of DFSOutputStream's writing concurrently, the number of
> outstanding packets could be large. The byte arrays created by those packets
> could occupy a lot of memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
