[
https://issues.apache.org/jira/browse/HIVE-30168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-30168:
----------------------------------
Labels: pull-request-available (was: )
> Docker: the standalone Tez AM starts without the glibc malloc arena cap
> -----------------------------------------------------------------------
>
> Key: HIVE-30168
> URL: https://issues.apache.org/jira/browse/HIVE-30168
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: pull-request-available
>
> The Hive docker image starts every service from
> packaging/src/docker/entrypoint.sh:
>
> - HiveServer2 and the metastore through bin/hive, which runs `hadoop jar`;
> bin/hadoop sources hadoop-config.sh, whose hadoop_os_tricks exports
> MALLOC_ARENA_MAX=4 on Linux
>
>
> - LLAP through runLlapDaemon.sh, which exports MALLOC_ARENA_MAX=4
>
>
> - the standalone Tez AM (run_tezam, HIVE-29419) with plain java: it runs
> with glibc's default limit of 8 malloc arenas per core
>
>
>
>
>
> Seen on Kubernetes (24 LLAP daemons, TPC-DS 10 TB external Parquet on S3):
> the AM container (6144M limit, -Xmx4G) was OOMKilled with the heap 20% live
> and 1568 threads; memory outside the heap filled the container.
>
>
>
>
> Fix: run_tezam defaults MALLOC_ARENA_MAX to 4, as the other services get
> it; a value set in the container's environment is kept.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
