Gopinathan A created HBASE-5881: ----------------------------------- Summary: BuiltIn Gzip compressor & decompressor not getting pooled, leading to native memory leak Key: HBASE-5881 URL: https://issues.apache.org/jira/browse/HBASE-5881 Project: HBase Issue Type: Bug Components: io Affects Versions: 0.92.1 Reporter: Gopinathan A Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0
This issue will occur only in hadoop 23.x & above/ In hadoop 0.20.x {code} public static void returnDecompressor(Decompressor decompressor) { if (decompressor == null) { return; } decompressor.reset(); payback(decompressorPool, decompressor); } {code} In hadoop 0.23.x {code} public static void returnDecompressor(Decompressor decompressor) { if (decompressor == null) { return; } // if the decompressor can't be reused, don't pool it. if (decompressor.getClass().isAnnotationPresent(DoNotPool.class)) { return; } decompressor.reset(); payback(decompressorPool, decompressor); } {code} Here annotation has been added. By default this library will be loaded if there are no native library. {code} @DoNotPool public class BuiltInGzipDecompressor {code} Due to this each time new compressor/decompressor will be loaded, this leads to native memory leak. {noformat} 2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz] 2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz] 2012-04-25 22:11:48,093 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.gz] {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira