Michael Han created ZOOKEEPER-3964:
--------------------------------------
Summary: Introduce RocksDB snap and implement change data capture
to enable incremental snapshot
Key: ZOOKEEPER-3964
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3964
Project: ZooKeeper
Issue Type: Sub-task
Components: rocksdb, server
Reporter: Michael Han
Assignee: Michael Han
This is the first step of enabling on disk storage engine for ZooKeeper by
extending the existing Snap interface and implement a RocksDB backed snapshot.
Comparing to file based snapshot, RocksDB based snapshot is superior for big in
memory data tree as it supports incremental snapshot by only serializing the
changed data between snapshots.
High level overview:
* Extend Snap interface so every thing that's need serialize has a presence on
the interface.
* Implement RocksDB based snapshot, and bidirectional conversations between
File based snapshot and RocksDB snapshot, for back / forward compatibility.
* Change data capture is implemented by buffering transactions applied to data
tree, and applied to RocksDB when processing each transaction. An incremental
snapshot thus only requires RocksDB flush. ZK will always do a full snapshot
when first loading the data tree during the start process.
* By default, this feature is disabled. Users need opt in by explicitly
specify a Java system property to instantiate RocksDBSnap at runtime.
This work is based on top of the patch attached to ZOOKEEPER-3783 (kudos to
Fangmin and co at FB), with some bug / test fixes and adjustment so it can
cleanly apply to master branch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
