[
https://issues.apache.org/jira/browse/TEZ-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated TEZ-1162:
--------------------------------
Fix Version/s: 0.4.1
> Tez leaks CodecPool buffers
> ---------------------------
>
> Key: TEZ-1162
> URL: https://issues.apache.org/jira/browse/TEZ-1162
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Gopal V
> Assignee: Gopal V
> Fix For: 0.5.0, 0.4.1
>
> Attachments: TEZ-1162.1.patch
>
>
> Tez Fetcher currently leaks a codec pool (~32kb) for each partition fetched
> This causes task failures due to the direct buffers allocated by the codec,
> which are not freed until a full GC (but whose allocation does not trigger
> GC).
> It is possible to perform the entire shuffle operations without ever
> triggering a full GC, the following case
> {code}
> Container exited with a non-zero exit code 143
> ], AttemptID:attempt_1399351577718_2330_1_05_000020_3 Info:Container
> container_1399351577718_2330_01_000310 COMPLETED with diagnostics set to
> [Container [pid=1734,containerID=container_1399351577718_2330_01_000310] is
> running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical
> memory used; 5.4 GB of 40 GB virtual memory used. Killing container.
> container_1399351577718_2330_01_000365/ $ grep -ri "CodecPool.*brand-new"
> syslog* | wc -l
> 6988
> {code}
> That is approx ~436Mb leak on a JVM spun up with -Xmx3500m & 4096m container.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
