[ https://issues.apache.org/jira/browse/PARQUET-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505934#comment-16505934 ]
ASF GitHub Bot commented on PARQUET-1320: ----------------------------------------- caneGuy opened a new pull request #492: PARQUET-1320: Fix potential direct memory leak for snappy decompressor URL: https://github.com/apache/parquet-mr/pull/492 When use NonBlockedDecompressorStream for decompress. If we do not get any full gc for old gen.we may failed by off-heap memory leak This patch fix this potential leak. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fast clean unused direct memory when decompress > ----------------------------------------------- > > Key: PARQUET-1320 > URL: https://issues.apache.org/jira/browse/PARQUET-1320 > Project: Parquet > Issue Type: Improvement > Components: parquet-mr > Reporter: zhoukang > Priority: Major > > When use *NonBlockedDecompressorStream* which call: > *SnappyDecompressor.setInput* > {code:java} > public synchronized void setInput(byte[] buffer, int off, int len) { > SnappyUtil > public synchronized void setInput(byte[] buffer, int off, int len) { > SnappyUtil.validateBuffer(buffer, off, len); > if (inputBuffer.capacity() - inputBuffer.position() < len) { > ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + > len); > inputBuffer.rewind(); > newBuffer.put(inputBuffer); > inputBuffer = newBuffer; > } else { > inputBuffer.limit(inputBuffer.position() + len); > } > inputBuffer.put(buffer, off, len); > } > .validateBuffer(buffer, off, len); > if (inputBuffer.capacity() - inputBuffer.position() < len) { > ByteBuffer newBuffer = ByteBuffer.allocateDirect(inputBuffer.position() + > len); > inputBuffer.rewind(); > newBuffer.put(inputBuffer); > inputBuffer = newBuffer; > } else { > inputBuffer.limit(inputBuffer.position() + len); > } > inputBuffer.put(buffer, off, len); > } > {code} > If we do not get any full gc for old gen.we may failed by off-heap memory leak > -- This message was sent by Atlassian JIRA (v7.6.3#76005)