[ https://issues.apache.org/jira/browse/HADOOP-16242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16813959#comment-16813959 ]
Da Zhou commented on HADOOP-16242: ---------------------------------- Attached some graphs of memory usage after adding buffer pool, in the test the memory usage is reduced by 30% after adding buffer pool. All tests passed using SharedKey: Tests run: 40, Failures: 0, Errors: 0, Skipped: 0 Tests run: 333, Failures: 0, Errors: 0, Skipped: 21 Tests run: 190, Failures: 0, Errors: 0, Skipped: 15 > ABFS: add bufferpool to AbfsOutputStream > ---------------------------------------- > > Key: HADOOP-16242 > URL: https://issues.apache.org/jira/browse/HADOOP-16242 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Affects Versions: 3.2.0 > Reporter: Da Zhou > Assignee: Da Zhou > Priority: Major > Attachments: HADOOP-16242-001.patch, > WritePerformanceWithBufferPool.pdf > > > In current AbfsOutPutStream, new temporary buffers always get allocated when > uploading the data to server. These big chunk creates huge pressure on the > GC, and in some extreme cases that GC doesn't happen in time(observed in some > customer's env), it also lead to OutOfMemory issue easily. > Hence we need to replace the current implementation with a bufferpool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org