Re: Question about Checkpoint Storage (RocksDB)

On Tue, Jul 26, 2016 at 2:15 PM, Sameer W wrote: > 1. Calling clear() on the KV state is only possible for snapshots right? Do > you control that for checkpoints too. What do you mean with snapshots vs. checkpoints exactly? > 2. Assuming that the user has no control over

Re: Question about Checkpoint Storage (RocksDB)

Thanks Ufuk, That was very helpful. But that raised a few more questions :-): 1. Calling clear() on the KV state is only possible for snapshots right? Do you control that for checkpoints too. 2. Assuming that the user has no control over the checkpoint process outside of controlling the

Re: Question about Checkpoint Storage (RocksDB)

On Mon, Jul 25, 2016 at 8:50 PM, Sameer W wrote: > The question is, if using really long windows (in hours) if the state of the > window gets very large over time, would size of the RocksDB get larger? > Would replication to HDFS start causing performance bottlenecks? Also

Question about Checkpoint Storage (RocksDB)

Hi, My understanding about the RocksDB state backend is as follows: When using a RocksDB state backend, it the checkpoints are backed up locally (to the TaskManager) using the backup feature of RocksDB by taking snapshots from RocksDB which are consistent read-only views on the RockDB database.