Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24761
to look at the new patch set (#2).
Change subject: IMPALA-15316: Support huge pages with aggressive decommit off
......................................................................
IMPALA-15316: Support huge pages with aggressive decommit off
Currently, running with tcmalloc_aggressive_memory_decommit=false
is incompatible with using madvise_huge_pages=true. Gperftools
TCMalloc can hold on to memory and potentially break up huge
pages. That can lead to deceptive memory usage metrics.
This changes the system allocator to use mmap for huge pages
when tcmalloc_aggressive_memory_decommit=false. This bypasses
malloc and avoids any issues with TCMalloc behavior. This does
not add a new startup flag. Instead, it is now possible to use
tcmalloc_aggressive_memory_decommit=false together with
madvise_huge_pages=true. Users can disable the mmap behavior
by using madvise_huge_pages=false.
This adds the buffer-pool.mmapped-bytes metric to track memory
allocated via mmap. This is included in the process memory
tracker.
This does not change the behavior when building with Google
TCMalloc, as it is already huge page aware.
Testing:
- Ran tests with tcmalloc_aggressive_memory_decommit=false
and madvise_huge_pages=true.
Change-Id: I13ac5f065104d5cf35efb5029ba3c4d482904096
---
M be/src/runtime/bufferpool/buffer-allocator-test.cc
M be/src/runtime/bufferpool/buffer-allocator.cc
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/buffer-pool.cc
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/free-list-test.cc
M be/src/runtime/bufferpool/system-allocator.cc
M be/src/runtime/bufferpool/system-allocator.h
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M bin/start-impala-cluster.py
M common/thrift/metrics.json
12 files changed, 93 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/24761/2
--
To view, visit http://gerrit.cloudera.org:8080/24761
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I13ac5f065104d5cf35efb5029ba3c4d482904096
Gerrit-Change-Number: 24761
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
