[
https://issues.apache.org/jira/browse/HIVE-30168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30168:
----------------------------------
Description:
The Hive docker image starts every service from
packaging/src/docker/entrypoint.sh:
- HiveServer2 and the metastore through bin/hive, which runs `hadoop jar`;
bin/hadoop sources hadoop-config.sh, whose hadoop_os_tricks exports
MALLOC_ARENA_MAX=4 on Linux
- LLAP through runLlapDaemon.sh, which exports MALLOC_ARENA_MAX=4
- the standalone Tez AM (run_tezam, HIVE-29419) with plain java: it runs with
glibc's default limit of 8 malloc arenas per core
Seen on Kubernetes (24 LLAP daemons, TPC-DS 10 TB external Parquet on S3):
the AM container (6144M limit, -Xmx4G) was OOMKilled with the heap 20% live and
1568 threads; memory outside the heap filled the container.
Fix: run_tezam defaults MALLOC_ARENA_MAX to 4, as the other services get it;
a value set in the container's environment is kept.
was:
HIVE-29419 added run_tezam to packaging/src/docker/entrypoint.sh. It starts
`org.apache.tez.dag.app.DAGAppMaster `with plain java (only `hadoop classpath`
goes through bin/hadoop, in a subshell), so the AM does not get the
`MALLOC_ARENA_MAX=4` that every other Hadoop process gets: Hadoop's launcher
scripts export it (hadoop-functions.sh hadoop_os_tricks: "Newer versions of
glibc use an arena memory allocator that causes virtual memory usage to
explode. This interacts badly with the many threads that we use in Hadoop"),
YARN passes it into every container (yarn.nodemanager.admin-env), and
runLlapDaemon.sh sets it for LLAP. HiveServer2 and the metastore start through
bin/hive and get it. Without it glibc allows 8 arenas per core.
Seen on a TPC-DS 10 TB run (24 LLAP daemons, external Parquet): the AM's 6144M
container was OOMKilled during a query with its 4G heap 20% live (no full GC)
and 1568 threads at peak; non-heap memory, not the heap, filled the container.
Fix: run_tezam defaults MALLOC_ARENA_MAX to 4 and exports it; a value set in
the container's environment is kept.
> Docker: the standalone Tez AM starts without the glibc malloc arena cap
> -----------------------------------------------------------------------
>
> Key: HIVE-30168
> URL: https://issues.apache.org/jira/browse/HIVE-30168
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
>
> The Hive docker image starts every service from
> packaging/src/docker/entrypoint.sh:
>
> - HiveServer2 and the metastore through bin/hive, which runs `hadoop jar`;
> bin/hadoop sources hadoop-config.sh, whose hadoop_os_tricks exports
> MALLOC_ARENA_MAX=4 on Linux
>
>
> - LLAP through runLlapDaemon.sh, which exports MALLOC_ARENA_MAX=4
>
>
> - the standalone Tez AM (run_tezam, HIVE-29419) with plain java: it runs
> with glibc's default limit of 8 malloc arenas per core
>
>
>
>
>
> Seen on Kubernetes (24 LLAP daemons, TPC-DS 10 TB external Parquet on S3):
> the AM container (6144M limit, -Xmx4G) was OOMKilled with the heap 20% live
> and 1568 threads; memory outside the heap filled the container.
>
>
>
>
> Fix: run_tezam defaults MALLOC_ARENA_MAX to 4, as the other services get
> it; a value set in the container's environment is kept.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
