Hello Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6471
to look at the new patch set (#2).
Change subject: log_block_manager: use sparse_hash_map for block map
......................................................................
log_block_manager: use sparse_hash_map for block map
Based on my testing, this reduces the memory usage of 1M blocks from
~24M to ~9MB. Lookup performance should be similar. Write performance
may be slightly slower but this is relatively infrequent and not on a
hot path.
I also did an allocation trace of a simple program putting 1M <int,int>
pairs into both types of maps. With unordered_map, this resulted in 1M
16-byte allocations and several larger allocations as the hashtable
bucket array grew (largest being 8MB). With the sparse_hash_map, the
allocation sizes were evenly distributed with ~55K allocations at a
bunch of sizes between 16 and 384 bytes, with no large allocations. This
should be more efficient in terms of avoiding memory fragmentation or
longer resizing pauses as blocks are added.
Change-Id: I2e75deba7f7fb8d4f800695f195304df603e4ce9
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
2 files changed, 6 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/71/6471/2
--
To view, visit http://gerrit.cloudera.org:8080/6471
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2e75deba7f7fb8d4f800695f195304df603e4ce9
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
