[
https://issues.apache.org/jira/browse/IMPALA-11753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy resolved IMPALA-11753.
----------------------------------------
Resolution: Fixed
> CatalogD OOMkilled due to natively allocated memory
> ---------------------------------------------------
>
> Key: IMPALA-11753
> URL: https://issues.apache.org/jira/browse/IMPALA-11753
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
> Labels: OOM
>
> We can hit this bug in several installations:
> https://bugs.openjdk.org/browse/JDK-8257032
> (another nice description of the issue can be found here:
> https://medium.com/swlh/native-memory-the-silent-jvm-killer-595913cba8e7)
> The problem is that we are creating our own Deflater object and pass it to
> the constructor of DeflaterOutputStream:
> https://github.com/apache/impala/blob/84fa6d210d3966e5ece8b4ac84ff8bd8780dec4e/fe/src/main/java/org/apache/impala/util/CompressionUtil.java#L47
> This means that Java's DeflaterOutputStream won't assume ownership on the
> Deflater, and won't invoke its end() method:
> *
> https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L144
> *
> https://github.com/openjdk/jdk/blob/a249a52501f3cd7d4fbe5293d14ac8d0d6ffcc69/src/java.base/share/classes/java/util/zip/DeflaterOutputStream.java#L246-L247
> The Deflater's methods are implemented in C and allocate native memory. This
> means that until the GC doesn't destroy the unreachable Deflater objects they
> can consume quite much native memory. In some scenarios it can even result in
> OOMKills by the kernel.
> We should solve this issue by either
> * override close() of DeflaterOutputStream (like mentioned in the above blog
> post)
> * directly invoke end() on the Deflater object in a finally clause
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
