Hello Alexey Serbin, Yuqi Du, Yifan Zhang, Kudu Jenkins, Abhishek Chennaka, 
KeDeng, Wang Xixu,

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18569

to look at the new patch set (#61).

Change subject: KUDU-3371 [fs] Use RocksDB to store LBM metadata
......................................................................

KUDU-3371 [fs] Use RocksDB to store LBM metadata

Since the LogBlockContainerNativeMeta stores block records
sequentially in the metadata file, the live blocks maybe
in a very low ratio, so it may cause serious disk space
amplification and long time bootstrap consumption.

This patch introduces a new class LogBlockContainerRdbMeta
which uses RocksDB to store LBM metadata, a new item will
be Put() into RocksDB when a new block is created in LBM,
and the item will be Delete() from RocksDB when the block
is removed from LBM. Data in RocksDB can be maintained by
RocksDB itself, i.e. deleted items will be GCed so it's not
needed to rewrite the metadata as how we do in
LogBlockContainerNativeMeta.

The implementation also reuses most logic of the base class
LogBlockContainer, the main different with
LogBlockContainerNativeMeta is LogBlockContainerRdbMeta
stores block records metadata in RocksDB rather than a
native file, the main implementation of interfaces from
the base clase including:
a. Create container
   Data file is created similar to LogBlockContainerNativeMeta,
   but the metadata part is stored in RocksDB with keys
   constructed as "<container_id>.<block_id>", and values are
   the same to the records stored in metadata file of
   LogBlockContainerNativeMeta.
b. Open container
   Similar to LogBlockContainerNativeMeta, and it's not needed
   to check the metadata part, because it has been checked when
   load containers when bootstrap.
c. Destroy container
   If the container is dead (full and no live blocks), remove
   the data file, and clean up metadata part, by deleting all
   the keys prefixed by "<container_id>".
d. Load container (by ProcessRecords())
   Iterate the RocksDB in the key range
   [<container_id>, <next_container_id>), because dead blocks
   have been deleted directly, thus only live block records
   will be populated, we can use them as LogBlockContainerNativeMeta.
e. Create blocks in a container
   Put() serialized BlockRecordPB records into RocksDB, keys
   are contructed the same to the above.
f. Remove blocks from a container
   Contruct the keys same to the above, Delete() them from RocksDB
   in batch.

This patch contains the following changes:
- Adds a new block manager type named 'logr', it use RocksDB
  to store LBM metadata, it is specified by flag
  '--block_manager'.
- Related tests add new parameterized value to test the case
  of "--block_manager=logr".

It's optional to use RocksDB, we can use the former LBM as
before, we will introduce more tools to convert data between
the two implementations in the future.

The optimization is obvious as shown in JIRA KUDU-3371, it
shows that reopen staged reduced upto 90% time cost.

Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f
---
M src/kudu/benchmarks/CMakeLists.txt
M src/kudu/client/CMakeLists.txt
M src/kudu/consensus/CMakeLists.txt
M src/kudu/fs/CMakeLists.txt
M src/kudu/fs/block_manager-stress-test.cc
M src/kudu/fs/block_manager-test.cc
M src/kudu/fs/block_manager.h
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/dir_manager.cc
M src/kudu/fs/dir_manager.h
M src/kudu/fs/dir_util.cc
M src/kudu/fs/file_block_manager.h
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_report.cc
M src/kudu/fs/fs_report.h
M src/kudu/fs/log_block_manager-test-util.cc
M src/kudu/fs/log_block_manager-test-util.h
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/dense_node-itest.cc
M src/kudu/integration-tests/ts_recovery-itest.cc
M src/kudu/server/CMakeLists.txt
M src/kudu/tablet/compaction-test.cc
M src/kudu/tools/CMakeLists.txt
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/util/CMakeLists.txt
M thirdparty/build-definitions.sh
31 files changed, 1,610 insertions(+), 166 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/18569/61
--
To view, visit http://gerrit.cloudera.org:8080/18569
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f
Gerrit-Change-Number: 18569
Gerrit-PatchSet: 61
Gerrit-Owner: Yingchun Lai <laiyingc...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <achenn...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <ale...@apache.org>
Gerrit-Reviewer: KeDeng <kdeng...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Wang Xixu <1450306...@qq.com>
Gerrit-Reviewer: Yifan Zhang <chinazhangyi...@163.com>
Gerrit-Reviewer: Yingchun Lai <laiyingc...@apache.org>
Gerrit-Reviewer: Yuqi Du <shenxingwuy...@gmail.com>

Reply via email to