[
https://issues.apache.org/jira/browse/HIVE-30168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30168:
----------------------------------
Description:
HIVE-29419 added run_tezam to packaging/src/docker/entrypoint.sh. It starts
org.apache.tez.dag.app.DAGAppMaster with
plain java (only `hadoop classpath` goes through bin/hadoop, in a subshell), so
the AM does not get the MALLOC_ARENA_MAX=4 that every other Hadoop process
gets: Hadoop's launcher scripts export it (hadoop-functions.sh
hadoop_os_tricks: "Newer versions of glibc use an arena memory allocator that
causes virtual memory usage to explode.
This interacts badly with the many threads that we use in Hadoop"), YARN passes
it into every container (yarn.nodemanager.admin-env), and runLlapDaemon.sh sets
it for LLAP. HiveServer2 and the metastore start through bin/hive and get it.
Without it glibc allows 8 arenas per core.
Seen on a TPC-DS 10 TB run (24 LLAP daemons, external Parquet): the AM's 6144M
container was OOMKilled during a query with its 4G heap 20% live (no full GC)
and 1568 threads at peak; non-heap memory, not the heap, filled the container.
Fix: run_tezam defaults MALLOC_ARENA_MAX to 4 and exports it; a value set in
the container's environment is kept.
> Docker: the standalone Tez AM starts without the glibc malloc arena cap
> -----------------------------------------------------------------------
>
> Key: HIVE-30168
> URL: https://issues.apache.org/jira/browse/HIVE-30168
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
>
> HIVE-29419 added run_tezam to packaging/src/docker/entrypoint.sh. It starts
> org.apache.tez.dag.app.DAGAppMaster with
> plain java (only `hadoop classpath` goes through bin/hadoop, in a subshell),
> so the AM does not get the MALLOC_ARENA_MAX=4 that every other Hadoop process
> gets: Hadoop's launcher scripts export it (hadoop-functions.sh
> hadoop_os_tricks: "Newer versions of glibc use an arena memory allocator that
> causes virtual memory usage to explode.
> This interacts badly with the many threads that we use in Hadoop"), YARN
> passes it into every container (yarn.nodemanager.admin-env), and
> runLlapDaemon.sh sets it for LLAP. HiveServer2 and the metastore start
> through bin/hive and get it. Without it glibc allows 8 arenas per core.
> Seen on a TPC-DS 10 TB run (24 LLAP daemons, external Parquet): the AM's
> 6144M container was OOMKilled during a query with its 4G heap 20% live (no
> full GC) and 1568 threads at peak; non-heap memory, not the heap, filled the
> container.
> Fix: run_tezam defaults MALLOC_ARENA_MAX to 4 and exports it; a value set in
> the container's environment is kept.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
