[jira] [Updated] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21162: --- Fix Version/s: (was: 1.5.0) > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch, > HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21162: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch, > HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21162: --- Attachment: HBASE-21162-branch-1.patch > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch, > HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21162: --- Attachment: (was: HBASE-21162-branch-1.patch) > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch, > HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21162: --- Attachment: HBASE-21162-branch-1.patch > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch, > HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21162: --- Attachment: HBASE-21162-branch-1.patch > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21162: --- Status: Patch Available (was: Open) > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21162: --- Attachment: HBASE-21162-branch-1.patch > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
