[ https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15492619#comment-15492619 ]
Duo Zhang commented on HBASE-16635: ----------------------------------- Ah after a careful review, I found some leaks... First is in SaslWrapHandler, I forgot to release the ByteBuf after read all its content to a byte array. Second, the two connection header ByteBufs in NettyRpcConnection. It is allocated by Unpooled so I ignored the release intentionally when writing down the code, maybe. But after reading the code of netty, I believe the ByteBufs allocated by Unpooled should also be released... I do not know why I have an illusion that an unpooled ByteBuf does not need to be released. I had hit a 'buffer already released' problem when I forgot to use retainedDuplicate before writing out the header. This definitely means netty will release unpooled ByteBuf... Please tell me if you have other findings. Thanks. My fault. > RpcClient under heavy load leaks some netty bytebuf > --------------------------------------------------- > > Key: HBASE-16635 > URL: https://issues.apache.org/jira/browse/HBASE-16635 > Project: HBase > Issue Type: Bug > Reporter: ramkrishna.s.vasudevan > Assignee: ramkrishna.s.vasudevan > Priority: Minor > Fix For: 2.0.0 > > > Yet to analyse the actual root cause. > But the case is that when we run a PE tool with 50 threads under heavy load > when the writes are clogged I think we have some netty Bytebuf leak. Not sure > if it is a serious issue but we get this log > {code} > 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16] > util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's > garbage-collected. Enable advanced leak reporting to find out where the leak > occurred. To enable advanced leak reporting, specify the JVM option > '-Dio.netty.leakDetection.level=advanced' or call > ResourceLeakDetector.setLevel() See > http://netty.io/wiki/reference-counted-objects.html for more information. > {code} > So reading the given link it is because of some ByteBuf that was not released > properly by the client and hence it gets GCed automatically. Netty provides > tips and tricks to find the root cause. Will get back here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)