Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5073: Part 1: add option to use mmap() for buffer pool ......................................................................
IMPALA-5073: Part 1: add option to use mmap() for buffer pool Support allocating with mmap instead of TCMalloc to give more control over memory usage. Also tell Linux to back larger buffers with huge pages when possible to reduce TLB pressure. The main complication is that memory returned by mmap() is not necessarily aligned to a huge page boundary, so we need to "fix up" the mapping ourselves. Adds additional memory metrics, since we previously relied on the assumption that all memory was allocated through TCMalloc. memory.total-used tracks the total across the buffer pool and TCMalloc. When the buffer pool is not present, they just report the TCMalloc values. This can be enabled with the --mmap_buffers flag. The transparent huge pages support can be disabled with the --madvise_huge_pages startup flag. At some point this should become the default, but it requires more work to validate perf and resource used (virtual address space, etc). Testing: Added some unit tests to test edge cases and the different supported flags. Many pre-existing tests also exercise the modified code. Change-Id: Ifbc748f74adcbbdcfa45f3ec7df98284925acbd6 Reviewed-on: http://gerrit.cloudera.org:8080/6474 Reviewed-by: Tim Armstrong <tarmstr...@cloudera.com> Tested-by: Impala Public Jenkins --- M be/src/catalog/catalogd-main.cc M be/src/runtime/bufferpool/buffer-allocator-test.cc M be/src/runtime/bufferpool/buffer-allocator.h M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/bufferpool/buffer-pool.h M be/src/runtime/bufferpool/system-allocator.cc M be/src/runtime/bufferpool/system-allocator.h M be/src/runtime/exec-env.cc M be/src/statestore/statestored-main.cc M be/src/util/memory-metrics.cc M be/src/util/memory-metrics.h M be/src/util/metrics-test.cc M be/src/util/metrics.h M common/thrift/generate_error_codes.py M common/thrift/metrics.json 15 files changed, 396 insertions(+), 41 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/6474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ifbc748f74adcbbdcfa45f3ec7df98284925acbd6 Gerrit-PatchSet: 13 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Dan Hecht <dhe...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
