[jira] [Comment Edited] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609544#comment-16609544 ] Andrew Purtell edited comment on HBASE-21162 at 9/10/18 5:30 PM: - bq. Since BoundedByteBufferPool is created just once in RpcServer, it shouldn't leak 40 gigs of native memory. We had a leak that went away when we disabled the RPC reservoir, so experimental observation counters your assertion. The change I pointed to is suspicious because in that change I replaced existing accounting with something using atomics, and now we have an experimentally confirmed leak from that code. [~shahrs87] We aren't leaking instances of the class boundedbytebufferpool, we are leaking direct buffers from the pool itself. Hope that helps your understanding of the issue. was (Author: apurtell): bq. Since BoundedByteBufferPool is created just once in RpcServer, it shouldn't leak 40 gigs of native memory. We had a leak that went away when we disabled the RPC reservoir, so experimental observation counters your assertion. The change I pointed to is suspicious because in that change I replaced existing accounting with something using atomics, and now we have an experimentally confirmed leak from that code. > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch, > HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609544#comment-16609544 ] Andrew Purtell edited comment on HBASE-21162 at 9/10/18 5:28 PM: - bq. Since BoundedByteBufferPool is created just once in RpcServer, it shouldn't leak 40 gigs of native memory. We had a leak that went away when we disabled the RPC reservoir, so experimental observation counters your assertion. The change I pointed to is suspicious because in that change I replaced existing accounting with something using atomics, and now we have an experimentally confirmed leak from that code. was (Author: apurtell): > Since BoundedByteBufferPool is created just once in RpcServer, it shouldn't > leak 40 gigs of native memory. We had a leak that went away when we disabled the RPC reservoir, so experimental observation counters your assertion. The change I pointed to is suspicious because in that change I replaced existing accounting with something using atomics, and now we have an experimentally confirmed leak from that code. > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch, > HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[
https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609518#comment-16609518
]
Rushabh S Shah edited comment on HBASE-21162 at 9/10/18 5:19 PM:
-
{quote}I think the change is suspicious and should be reverted.
{quote}
[~apurtell] thanks for the reply. Just for my knowledge why do you think this
change is suspicious ? Since {{BoundedByteBufferPool}} is created just once in
{{RpcServer, it shouldn't leak 40 gigs of native memory.}}
was (Author: shahrs87):
{quote}I think the change is suspicious and should be reverted.
{quote}
Byte[~apurtell] thanks for the reply. Just for my knowledge why do you think
this change is suspicious ? Since {{BoundedByteBufferPool}} is created just
once in {{RpcServer, it shouldn't leak 40 gigs of native memory.}}
> Revert suspicious change to BoundedByteBufferPool and disable use of direct
> buffers for IPC reservoir by default
>
>
> Key: HBASE-21162
> URL: https://issues.apache.org/jira/browse/HBASE-21162
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch,
> HBASE-21162-branch-1.patch
>
>
> We had a production incident where we traced the issue to a direct buffer
> leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false
> and after that no native memory leak could be observed in any regionserver
> process under the triggering load.
> On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to
> BoundedByteBufferPool that is suspicious given this finding. It was committed
> to branch-1.4 and branch-1. I'm going to revert this change.
> In addition the allocation of direct memory for the server RPC reservoir is a
> bit problematic in that tracing native memory or direct buffer leaks to a
> particular class or compilation unit is difficult, so I also propose
> allocating the reservoir on the heap by default instead. Should there be a
> leak it is much easier to do an analysis of a heap dump with familiar tools
> to find it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[
https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609518#comment-16609518
]
Rushabh S Shah edited comment on HBASE-21162 at 9/10/18 5:18 PM:
-
{quote}I think the change is suspicious and should be reverted.
{quote}
Byte[~apurtell] thanks for the reply. Just for my knowledge why do you think
this change is suspicious ? Since {{BoundedByteBufferPool}} is created just
once in {{RpcServer, it shouldn't leak 40 gigs of native memory.}}
was (Author: shahrs87):
{quote}I think the change is suspicious and should be reverted.
{quote}
Byte[~apurtell] thanks for the reply. Just for my knowledge why do you think
this change is suspicious ?
Also \{{BoundedByteBufferPool}} is created just once in \{{RpcServer}}.
> Revert suspicious change to BoundedByteBufferPool and disable use of direct
> buffers for IPC reservoir by default
>
>
> Key: HBASE-21162
> URL: https://issues.apache.org/jira/browse/HBASE-21162
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.7
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 1.5.0, 1.4.8
>
> Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch,
> HBASE-21162-branch-1.patch
>
>
> We had a production incident where we traced the issue to a direct buffer
> leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false
> and after that no native memory leak could be observed in any regionserver
> process under the triggering load.
> On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to
> BoundedByteBufferPool that is suspicious given this finding. It was committed
> to branch-1.4 and branch-1. I'm going to revert this change.
> In addition the allocation of direct memory for the server RPC reservoir is a
> bit problematic in that tracing native memory or direct buffer leaks to a
> particular class or compilation unit is difficult, so I also propose
> allocating the reservoir on the heap by default instead. Should there be a
> leak it is much easier to do an analysis of a heap dump with familiar tools
> to find it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16607580#comment-16607580 ] Andrew Purtell edited comment on HBASE-21162 at 9/7/18 7:39 PM: [~mdrob] Latest patch suppresses warning for http://errorprone.info/bugpattern/NonAtomicVolatileUpdate in BoundedByteBufferPool; otherwise unchanged Edit: Confirmed with a compile of hbase-common with -PerrorProne was (Author: apurtell): [~mdrob] Latest patch suppresses warning for http://errorprone.info/bugpattern/NonAtomicVolatileUpdate in BoundedByteBufferPool; otherwise unchanged > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch, HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21162) Revert suspicious change to BoundedByteBufferPool and disable use of direct buffers for IPC reservoir by default
[ https://issues.apache.org/jira/browse/HBASE-21162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606582#comment-16606582 ] Andrew Purtell edited comment on HBASE-21162 at 9/7/18 12:49 AM: - I think the change is suspicious and should be reverted. Even if we are not allocating from direct buffers we could still leak heap was (Author: apurtell): We want the reservoir for its original purpose of avoiding allocations per RPC, so it should be enabled; but as I suggest not allocated via direct buffers by default. > Revert suspicious change to BoundedByteBufferPool and disable use of direct > buffers for IPC reservoir by default > > > Key: HBASE-21162 > URL: https://issues.apache.org/jira/browse/HBASE-21162 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 1.5.0, 1.4.8 > > Attachments: HBASE-21162-branch-1.patch > > > We had a production incident where we traced the issue to a direct buffer > leak. On a hunch we tried setting hbase.ipc.server.reservoir.enabled = false > and after that no native memory leak could be observed in any regionserver > process under the triggering load. > On HBASE-19239 (Fix findbugs and error-prone issues) I made a change to > BoundedByteBufferPool that is suspicious given this finding. It was committed > to branch-1.4 and branch-1. I'm going to revert this change. > In addition the allocation of direct memory for the server RPC reservoir is a > bit problematic in that tracing native memory or direct buffer leaks to a > particular class or compilation unit is difficult, so I also propose > allocating the reservoir on the heap by default instead. Should there be a > leak it is much easier to do an analysis of a heap dump with familiar tools > to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
