Thank you. That clears it up. I meant SavePoints. Sorry I used the term Snapshots in its place :-).
Thanks, Sameer On Tue, Jul 26, 2016 at 8:33 AM, Ufuk Celebi <u...@apache.org> wrote: > On Tue, Jul 26, 2016 at 2:15 PM, Sameer W <sam...@axiomine.com> wrote: > > 1. Calling clear() on the KV state is only possible for snapshots right? > Do > > you control that for checkpoints too. > > What do you mean with snapshots vs. checkpoints exactly? > > > 2. Assuming that the user has no control over the checkpoint process > outside > > of controlling the checkpoint interval , when is the RocksDB cleared of > the > > operator state for checkpoints after they are long past. It seems like > there > > are only two checkpoints that are really necessary to maintain, the > current > > one and the previous one for restore. Does Flink clean up checkpoints on > a > > timer? When it does clean up checkpoints does it also clean up the state > > backend (I am assuming they are different). > > Yes, here: > https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/fault_tolerance.html > > By default, only one completed checkpoint is kept. > > > 3. The pre-aggregating windows was very helpful as the WindowFunction is > now > > passed the pre-aggregated state. For windows, are the Reduce and Fold > > functions called on each element event before the window is triggered. I > can > > see how that would work where the pre-compute is done per element but the > > actual output is emitted only when the window is fired. But that is only > > possible if there are no Evictors defined on the window? Also how are the > > elements fed to the Reduce/Fold function. Is it like MapReduce where > even if > > you are using a Iterator, in reality all the values for a key are not > > buffered into memory? Which ties back to how is RocksDB is used to store > a > > large window state before it is triggered. If my elements are > accumulating > > in a window (serving a ReduceFunction) does it spill to disk (RocksDB?) > when > > a threshold size is reached? > > - The function is called before adding the element to the window KV state > - Yes, only possible if no evictors are defined > - The window reduce function is applied directly on the elements of > the stream and then update the KvState instance (e.g. update RocksDB) > - Operations with RocksDB always touch RocksDB, which takes care of > spilling etc. >
