[ https://issues.apache.org/jira/browse/HDFS-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183623#comment-14183623 ]
Tsz Wo Nicholas Sze commented on HDFS-7276: ------------------------------------------- [~stack], thanks for taking a look. bq. The patch does not seem to be concerned with the amount of memory allocated since it only counts buffers, not how much has been actually allocated. Or is the idea throttle array allocations as means of throttling number of Packets outstanding? Yes, limiting the number of arrays also limits the number of outstanding packets. A full packet is around 64kB. We indeed only limit the outstanding full packets. bq. Any idea how much the synchronize on allocation/recycle and count as well as the wait/notifyAll changes write perf? I will do some performance tests. bq. Why volatile in below when doesn't seem to ever change (Should it be configurable? Would be good if MAX_ARRAYS was configurable at least for clients who'd like to postpone blocking): ... It needs volatile. Otherwise, other threads may not see the change done by reset(..). Configurable is a good idea. Let me think about how to do it. bq. Nit: IMO 1 << 11 is cryptic. Suggest instead you write out 2048? Sure, will do. > Limit the number of byte arrays used by DFSOutputStream > ------------------------------------------------------- > > Key: HDFS-7276 > URL: https://issues.apache.org/jira/browse/HDFS-7276 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client > Reporter: Tsz Wo Nicholas Sze > Assignee: Tsz Wo Nicholas Sze > Attachments: h7276_20141021.patch, h7276_20141022.patch, > h7276_20141023.patch, h7276_20141024.patch > > > When there are a lot of DFSOutputStream's writing concurrently, the number of > outstanding packets could be large. The byte arrays created by those packets > could occupy a lot of memory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)