[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606559#comment-16606559 ]
Rushabh S Shah commented on HBASE-21162: ---------------------------------------- Thanks [~apurtell] for creating the issue and uploading the patch. I was also watching the incident and can also verify that after setting property {{hbase.ipc.server.reservoir.direct.buffer}} to false, we no longer saw native memory leak. Regarding the patch, do we need to do any changes to {{BoundedByteBufferPool}}. Shouldn't just turning the config off be sufficient ? > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > ---------------------------------------------------------------------------------------------------------------- > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug > Affects Versions: 1.4.7 > Reporter: Andrew Purtell > Assignee: Andrew Purtell > Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)