You might also consider support for a Bigtable backend: HBase/Accumulo/Cassandra. The data model should be similar (identical?) to RocksDB and you get HA, recoverability, and support for really large state "for free".
On Thursday, June 9, 2016, Chen Qin <qinnc...@gmail.com> wrote: > Hi there, > > What is progress on incremental checkpointing? Does flink dev has plan to > work on this or JIRA to track this? super interested to know. > > I also research and consider use rocksdbstatebackend without running HDFS > cluster nor talk to S3. Some primitive idea is to use ZK to store / notify > state propagation progress and propagate via implement chain replication on > top of YARN provisioned storage node. > > Thanks, > Chen >
