On Tue, May 10, 2016 at 5:07 PM, Chen Qin <qinnc...@gmail.com> wrote:
> Future, to keep large key/value space, wiki point out using rocksdb as
> backend. My understanding is using rocksdb will write to local file systems
> instead of sync to s3. Does flink support memory->rocksdb(local disk)->s3
> checkpoint state split yet? Or would implement kvstate interface makes flink
> take care of large state problem?
Hey Chen,
when you use RocksDB, you only need to explicitly configure the file
system checkpoint directory, for which you can use S3:
new RocksDBStateBackend(new URI("s3://..."))
The local disk path are configured via the general Flink temp
directory configuration (see taskmanager.tmp.dirs in
https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html,
default is /tmp).
State is written to the local RocksDB instance and the RocksDB files
are copied to S3 on checkpoints.
Does this help?
– Ufuk
