[jira] [Commented] (HADOOP-18383) Codecs with @DoNotPool annotation are not closed causing memory leak

[ASF GitHub Bot (Jira)]

    [ 
https://issues.apache.org/jira/browse/HADOOP-18383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17579267#comment-17579267
 ] 

ASF GitHub Bot commented on HADOOP-18383:
-----------------------------------------

kevins-29 opened a new pull request, #4739:
URL: https://github.com/apache/hadoop/pull/4739

   ### Description of PR
   Explicitly call `end()` when returning `Decompressor` implementations with 
`DoNotPool` annotation to the `CodecPool`.
   
   ### How was this patch tested?
   I created the following 
[project](https://github.com/kevins-29/hadoop-gzip-memory-leak) to demo the 
leak. You can run the demo with 
   
   ``` shell
   ./gradlew run --args=1024
   ``` 
   
   and then monitor the memory usage using
   
   ```shell
   while true; do echo \"$(date +%Y-%m-%d' '%H:%M:%S)\",$(pmap -x <PID> | grep 
"total kB" | awk '{print $4}'); sleep 10; done;
   ```
   
   ### Results - Before Patch
   
   ```
   ...
   ```
   
   ### Results - After Patch
   ```
   ...
   ```
   
   
   




> Codecs with @DoNotPool annotation are not closed causing memory leak
> --------------------------------------------------------------------
>
>                 Key: HADOOP-18383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: common
>    Affects Versions: 3.3.2
>            Reporter: Kevin Sewell
>            Assignee: Kevin Sewell
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.4.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Compressors and Decompressions with a @DoNotPool annotation are not closed 
> when they are returned to the CodecPool, which causes a native memory leak.
>  
> I have included a link to a [Demo 
> Project|https://github.com/kevins-29/hadoop-gzip-memory-leak] demonstrating 
> the leak



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org