Hello Joe McDonnell, Bikramjeet Vig, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12262
to look at the new patch set (#3).
Change subject: IMPALA-7941: part 2/2: use cgroups memory limit
......................................................................
IMPALA-7941: part 2/2: use cgroups memory limit
This uses the functionality from part 1 to detect the CGroups memory
limit and use it to set a lower process memory limit if needed.
min(system memory, cgroups memory limit) is used instead of
system memory to determine the memory limit. Behaviour of
processes without a memory limit set via CGroups is unchanged.
The default behaviour of using 80% of the memory limit detected
is still in effect. This seems like an OK default, but may
lead to some amount of wasted memory.
Modify containers to have a default JVM heap size of 2GB and
--mem_limit_includes_jvm, so that the automatically configured
memory limit makes more sense.
start-impala-cluster.py is modified to exercise all of this.
Testing:
Tested a containerised cluster manually on my system, which has
32GB of RAM. Here's the breakdown from the memz/ page showing
the JVM heap and auto-configured memory limit.
Process: Limit=7.31 GB Total=1.94 GB Peak=1.94 GB
JVM: max heap size: Total=1.78 GB
JVM: non-heap committed: Total=35.56 MB
Buffer Pool: Free Buffers: Total=0
Buffer Pool: Clean Pages: Total=0
Buffer Pool: Unused Reservation: Total=0
Control Service Queue: Limit=50.00 MB Total=0 Peak=0
Data Stream Service Queue: Limit=374.27 MB Total=0 Peak=0
Data Stream Manager Early RPCs: Total=0 Peak=0
TCMalloc Overhead: Total=12.20 MB
Untracked Memory: Total=121.31 MB
Change-Id: Ie9fb4fb936a46fc194a204391d03c07c8c7fba21
---
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M bin/start-impala-cluster.py
M docker/coord_exec/Dockerfile
M docker/coordinator/Dockerfile
M docker/daemon_entrypoint.sh
M docker/executor/Dockerfile
7 files changed, 126 insertions(+), 75 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/12262/3
--
To view, visit http://gerrit.cloudera.org:8080/12262
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie9fb4fb936a46fc194a204391d03c07c8c7fba21
Gerrit-Change-Number: 12262
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
