[jira] [Commented] (HBASE-22963) Netty ByteBuf leak in rpc client implementation
[
https://issues.apache.org/jira/browse/HBASE-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925253#comment-16925253
]
Hudson commented on HBASE-22963:
Results for branch branch-2
[build #2238 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2238/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2238//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2238//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2238//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Netty ByteBuf leak in rpc client implementation
> ---
>
> Key: HBASE-22963
> URL: https://issues.apache.org/jira/browse/HBASE-22963
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> {noformat}
> 2019-08-31 21:48:27,067 ERROR [RS-EventLoopGroup-6-3]
> util.ResourceLeakDetector(317): LEAK: ByteBuf.release() was not called before
> it's garbage-collected. See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> org.apache.hbase.thirdparty.io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
>
> org.apache.hbase.thirdparty.io.netty.buffer.Unpooled.directBuffer(Unpooled.java:125)
>
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.(NettyRpcConnection.java:96)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:74)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:40)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:364)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:433)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$300(AbstractRpcClient.java:97)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$RpcChannelImplementation.callMethod(AbstractRpcClient.java:605)
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$Stub.scan(ClientProtos.java:42345)
>
> org.apache.hadoop.hbase.client.AsyncClientScanner.callOpenScanner(AsyncClientScanner.java:152)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.call(AsyncSingleRequestRpcRetryingCaller.java:82)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.lambda$doCall$7(AsyncSingleRequestRpcRetryingCaller.java:115)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.AsyncRegionLocator.lambda$getRegionLocation$3(AsyncRegionLocator.java:124)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ConnectionUtils.lambda$getOrFetch$6(ConnectionUtils.java:573)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Co
[jira] [Commented] (HBASE-22963) Netty ByteBuf leak in rpc client implementation
[
https://issues.apache.org/jira/browse/HBASE-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925246#comment-16925246
]
Hudson commented on HBASE-22963:
Results for branch branch-2.2
[build #585 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/585/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/585//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/585//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/585//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Netty ByteBuf leak in rpc client implementation
> ---
>
> Key: HBASE-22963
> URL: https://issues.apache.org/jira/browse/HBASE-22963
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> {noformat}
> 2019-08-31 21:48:27,067 ERROR [RS-EventLoopGroup-6-3]
> util.ResourceLeakDetector(317): LEAK: ByteBuf.release() was not called before
> it's garbage-collected. See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> org.apache.hbase.thirdparty.io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
>
> org.apache.hbase.thirdparty.io.netty.buffer.Unpooled.directBuffer(Unpooled.java:125)
>
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.(NettyRpcConnection.java:96)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:74)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:40)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:364)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:433)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$300(AbstractRpcClient.java:97)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$RpcChannelImplementation.callMethod(AbstractRpcClient.java:605)
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$Stub.scan(ClientProtos.java:42345)
>
> org.apache.hadoop.hbase.client.AsyncClientScanner.callOpenScanner(AsyncClientScanner.java:152)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.call(AsyncSingleRequestRpcRetryingCaller.java:82)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.lambda$doCall$7(AsyncSingleRequestRpcRetryingCaller.java:115)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.AsyncRegionLocator.lambda$getRegionLocation$3(AsyncRegionLocator.java:124)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ConnectionUtils.lambda$getOrFetch$6(ConnectionUtils.java:573)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFi
[jira] [Commented] (HBASE-22963) Netty ByteBuf leak in rpc client implementation
[
https://issues.apache.org/jira/browse/HBASE-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925245#comment-16925245
]
Hudson commented on HBASE-22963:
Results for branch branch-2.1
[build #1574 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1574/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1574//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1574//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1574//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Netty ByteBuf leak in rpc client implementation
> ---
>
> Key: HBASE-22963
> URL: https://issues.apache.org/jira/browse/HBASE-22963
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> {noformat}
> 2019-08-31 21:48:27,067 ERROR [RS-EventLoopGroup-6-3]
> util.ResourceLeakDetector(317): LEAK: ByteBuf.release() was not called before
> it's garbage-collected. See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> org.apache.hbase.thirdparty.io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
>
> org.apache.hbase.thirdparty.io.netty.buffer.Unpooled.directBuffer(Unpooled.java:125)
>
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.(NettyRpcConnection.java:96)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:74)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:40)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:364)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:433)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$300(AbstractRpcClient.java:97)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$RpcChannelImplementation.callMethod(AbstractRpcClient.java:605)
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$Stub.scan(ClientProtos.java:42345)
>
> org.apache.hadoop.hbase.client.AsyncClientScanner.callOpenScanner(AsyncClientScanner.java:152)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.call(AsyncSingleRequestRpcRetryingCaller.java:82)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.lambda$doCall$7(AsyncSingleRequestRpcRetryingCaller.java:115)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.AsyncRegionLocator.lambda$getRegionLocation$3(AsyncRegionLocator.java:124)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ConnectionUtils.lambda$getOrFetch$6(ConnectionUtils.java:573)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.
[jira] [Commented] (HBASE-22963) Netty ByteBuf leak in rpc client implementation
[
https://issues.apache.org/jira/browse/HBASE-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925224#comment-16925224
]
Hudson commented on HBASE-22963:
Results for branch master
[build #1412 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1412/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1412//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1412//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1412//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Netty ByteBuf leak in rpc client implementation
> ---
>
> Key: HBASE-22963
> URL: https://issues.apache.org/jira/browse/HBASE-22963
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
>
> {noformat}
> 2019-08-31 21:48:27,067 ERROR [RS-EventLoopGroup-6-3]
> util.ResourceLeakDetector(317): LEAK: ByteBuf.release() was not called before
> it's garbage-collected. See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> org.apache.hbase.thirdparty.io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
>
> org.apache.hbase.thirdparty.io.netty.buffer.Unpooled.directBuffer(Unpooled.java:125)
>
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.(NettyRpcConnection.java:96)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:74)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:40)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:364)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:433)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$300(AbstractRpcClient.java:97)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$RpcChannelImplementation.callMethod(AbstractRpcClient.java:605)
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$Stub.scan(ClientProtos.java:42345)
>
> org.apache.hadoop.hbase.client.AsyncClientScanner.callOpenScanner(AsyncClientScanner.java:152)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.call(AsyncSingleRequestRpcRetryingCaller.java:82)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.lambda$doCall$7(AsyncSingleRequestRpcRetryingCaller.java:115)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.AsyncRegionLocator.lambda$getRegionLocation$3(AsyncRegionLocator.java:124)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ConnectionUtils.lambda$getOrFetch$6(ConnectionUtils.java:573)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableF
[jira] [Commented] (HBASE-22963) Netty ByteBuf leak in rpc client implementation
[
https://issues.apache.org/jira/browse/HBASE-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921238#comment-16921238
]
Duo Zhang commented on HBASE-22963:
---
OK, I think the problem is cancelConnections...
Let me provide a patch...
> Netty ByteBuf leak in rpc client implementation
> ---
>
> Key: HBASE-22963
> URL: https://issues.apache.org/jira/browse/HBASE-22963
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Major
>
> {noformat}
> 2019-08-31 21:48:27,067 ERROR [RS-EventLoopGroup-6-3]
> util.ResourceLeakDetector(317): LEAK: ByteBuf.release() was not called before
> it's garbage-collected. See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> org.apache.hbase.thirdparty.io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
>
> org.apache.hbase.thirdparty.io.netty.buffer.Unpooled.directBuffer(Unpooled.java:125)
>
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.(NettyRpcConnection.java:96)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:74)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:40)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:364)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:433)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$300(AbstractRpcClient.java:97)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$RpcChannelImplementation.callMethod(AbstractRpcClient.java:605)
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$Stub.scan(ClientProtos.java:42345)
>
> org.apache.hadoop.hbase.client.AsyncClientScanner.callOpenScanner(AsyncClientScanner.java:152)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.call(AsyncSingleRequestRpcRetryingCaller.java:82)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.lambda$doCall$7(AsyncSingleRequestRpcRetryingCaller.java:115)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.AsyncRegionLocator.lambda$getRegionLocation$3(AsyncRegionLocator.java:124)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ConnectionUtils.lambda$getOrFetch$6(ConnectionUtils.java:573)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ZKAsyncRegistry.tryComplete(ZKAsyncRegistry.java:123)
>
> org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$1(ZKAsyncRegistry.java:165)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAn
[jira] [Commented] (HBASE-22963) Netty ByteBuf leak in rpc client implementation
[
https://issues.apache.org/jira/browse/HBASE-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921146#comment-16921146
]
Duo Zhang commented on HBASE-22963:
---
Looked at the code, seems in all the code path we will call cleanupConnection
before dereference the NettyRpcConnection...
Maybe something wrong when shutting down the rpc client. Let me dig more.
> Netty ByteBuf leak in rpc client implementation
> ---
>
> Key: HBASE-22963
> URL: https://issues.apache.org/jira/browse/HBASE-22963
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Major
>
> {noformat}
> 2019-08-31 21:48:27,067 ERROR [RS-EventLoopGroup-6-3]
> util.ResourceLeakDetector(317): LEAK: ByteBuf.release() was not called before
> it's garbage-collected. See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:
> Created at:
>
> org.apache.hbase.thirdparty.io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
>
> org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
>
> org.apache.hbase.thirdparty.io.netty.buffer.Unpooled.directBuffer(Unpooled.java:125)
>
> org.apache.hadoop.hbase.ipc.NettyRpcConnection.(NettyRpcConnection.java:96)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:74)
>
> org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:40)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:364)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:433)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$300(AbstractRpcClient.java:97)
>
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$RpcChannelImplementation.callMethod(AbstractRpcClient.java:605)
>
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$Stub.scan(ClientProtos.java:42345)
>
> org.apache.hadoop.hbase.client.AsyncClientScanner.callOpenScanner(AsyncClientScanner.java:152)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.call(AsyncSingleRequestRpcRetryingCaller.java:82)
>
> org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.lambda$doCall$7(AsyncSingleRequestRpcRetryingCaller.java:115)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.AsyncRegionLocator.lambda$getRegionLocation$3(AsyncRegionLocator.java:124)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ConnectionUtils.lambda$getOrFetch$6(ConnectionUtils.java:573)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> org.apache.hadoop.hbase.client.ZKAsyncRegistry.tryComplete(ZKAsyncRegistry.java:123)
>
> org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$1(ZKAsyncRegistry.java:165)
>
> org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
>
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
>
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> java.util.concurrent.Co
