Hello Alexey Serbin, Kudu Jenkins, Abhishek Chennaka, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18569 to look at the new patch set (#55). Change subject: WIP KUDU-3371 [fs] Use RocksDB to store LBM metadata ...................................................................... WIP KUDU-3371 [fs] Use RocksDB to store LBM metadata Since LogBlockContainer store block records sequentially in metadata file, the live blocks maybe in a very low ratio, and it cause disk space amplification and long time bootstrap. This patch use RocksDB to store LBM metadata, a new item will be Put() into RocksDB when a new block is created in LBM, and the item will be Delete() from RocksDB when the block is removed from LBM. Data in RocksDB can be maintained in RocksDB itself, i.e. deleted items will be GCed so doesn't need rewriting as how we do it in current LBM. The implemention also reuse most logic of LBM, the main difference is store Block records metadata in RocksDB. Introduce a new class LogrBlockManager that stores metadata in RocksDB, the main idea: a. Create container Data file is created as before, metadata is stored in keys prefixed by the container's id, append the block id, e.g. <container_id>.<block_id>. Make sure there is no such keys in RocksDB before this container created. b. Open container Make sure the data file is healthy. c. Deconstruct container If the container is dead (full and no live blocks), remove the data file, and clean up keys prefixed by the container's id. d. Load container (by ProcessRecords()) Iterate the RocksDB in the key range [<container_id>, <next_container_id>), only live block records will be populated, we can use them as before. e. Create blocks in a container Put() serialized BlockRecordPB records into RocksDB in batch, keys are in form of '<container_id>.<block_id>' as mentioned above. f. Remove blocks from a container Contruct the keys by container's id and block's id, Delete() them from RocksDB in batch. This patch contains the following changes: - Adds a new block manager type named 'logr', it use RocksDB to store LBM metadata, it is also specified by flag '--block_manager'. - block_manager-test supports to test LogrBlockManager - block_manager-stress-test supports to test LogrBlockManager - log_block_manager-test supports to test LogrBlockManager - tablet_server-test supports to test LogrBlockManager - dense_node-itest supports to test LogrBlockManager - kudu-tool-test supports to test LogrBlockManager It's optional to use RocksDB, we can use the former LBM as before, we can introduce more tools to convert data between the two implemention in the future. The optimization is obvious as shown in JIRA KUDU-3371, it shows that reopen staged reduced upto 90% time cost. Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f --- M src/kudu/benchmarks/CMakeLists.txt M src/kudu/client/CMakeLists.txt M src/kudu/consensus/CMakeLists.txt M src/kudu/fs/CMakeLists.txt M src/kudu/fs/block_manager-stress-test.cc M src/kudu/fs/block_manager-test.cc M src/kudu/fs/block_manager.h M src/kudu/fs/data_dirs.cc M src/kudu/fs/dir_manager.cc M src/kudu/fs/dir_manager.h M src/kudu/fs/dir_util.cc M src/kudu/fs/file_block_manager.h M src/kudu/fs/fs_manager-test.cc M src/kudu/fs/fs_manager.cc M src/kudu/fs/fs_report.cc M src/kudu/fs/fs_report.h M src/kudu/fs/log_block_manager-test-util.cc M src/kudu/fs/log_block_manager-test-util.h M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/dense_node-itest.cc M src/kudu/integration-tests/ts_recovery-itest.cc M src/kudu/server/CMakeLists.txt M src/kudu/tablet/compaction-test.cc M src/kudu/tools/CMakeLists.txt M src/kudu/tools/kudu-tool-test.cc M src/kudu/tserver/tablet_server-test.cc M src/kudu/util/CMakeLists.txt M thirdparty/build-definitions.sh 31 files changed, 1,591 insertions(+), 163 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/18569/55 -- To view, visit http://gerrit.cloudera.org:8080/18569 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f Gerrit-Change-Number: 18569 Gerrit-PatchSet: 55 Gerrit-Owner: Yingchun Lai <laiyingc...@apache.org> Gerrit-Reviewer: Abhishek Chennaka <achenn...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <ale...@apache.org> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai <laiyingc...@apache.org>