Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24761
Change subject: IMPALA-15316: Support huge pages with aggressive decommit off ...................................................................... IMPALA-15316: Support huge pages with aggressive decommit off Currently, running with tcmalloc_aggressive_memory_decommit=false is incompatible with using madvise_huge_pages=true. Gperftools TCMalloc can hold on to memory and potentially break up huge pages. That can lead to deceptive memory usage metrics. This changes the system allocator to use mmap for huge pages when tcmalloc_aggressive_memory_decommit=false. This bypasses malloc and avoids any issues with TCMalloc behavior. This does not add a new startup flag. Instead, it is now possible to use tcmalloc_aggressive_memory_decommit=false together with madvise_huge_pages=true. Users can disable the mmap behavior by using madvise_huge_pages=false. This adds the buffer-pool.mmapped-bytes metric to track memory allocated via mmap. This is included in the process memory tracker. This does not change the behavior when building with Google TCMalloc, as it is already huge page aware. Testing: - Ran tests with tcmalloc_aggressive_memory_decommit=false and madvise_huge_pages=true. Change-Id: I13ac5f065104d5cf35efb5029ba3c4d482904096 --- M be/src/runtime/bufferpool/buffer-allocator.cc M be/src/runtime/bufferpool/buffer-allocator.h M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/runtime/bufferpool/system-allocator.cc M be/src/runtime/bufferpool/system-allocator.h M be/src/util/memory-metrics.cc M be/src/util/memory-metrics.h M bin/start-impala-cluster.py M common/thrift/metrics.json 10 files changed, 81 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/24761/1 -- To view, visit http://gerrit.cloudera.org:8080/24761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I13ac5f065104d5cf35efb5029ba3c4d482904096 Gerrit-Change-Number: 24761 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <[email protected]>
