Hi, Vishal. IIUC, 1. FIFO compaction drops the old data by the configured size in L0, so the old data may be dropped but we could not know. That's why "it's basically a TTL compaction style and It is suited for keeping event log data with very low overhead (query log for example)". If it's the user case acceptable, I think the normal get and update is fine. 2. while triggering a checkpoint, Flink will trigger a forceful flush for memtable so all values retained in memory sync at the moment. It's unrelated to the compaction style (In addition to the deletion strategy of FIFO compaction). 3. We could also make FIFO compaction enable compaction to improve the query performance which may change the write amplification. So SST files will be compacted into larger SST files. I may miss your point about the problem you mean ?
Best, Hangxiang. On Tue, Jul 5, 2022 at 8:28 AM Zhanghao Chen <zhanghao.c...@outlook.com> wrote: > Hi Vishal, > > FIFO compaction with RocksDB can result in data loss as it discards the > oldest SST file by a size-based trigger or based on TTL (an internal > RocksDB option, irrelevant to the TTL setting in Flink). So use with cause. > > I'm not sure why you observed SST file compactions, did you manually > enabled allow compaction option for FIFO? > > Best, > Zhanghao Chen > ------------------------------ > *From:* Alexander Fedulov <alexan...@ververica.com> > *Sent:* Tuesday, July 5, 2022 5:49 > *To:* Vishal Surana <vis...@moengage.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Can FIFO compaction with RocksDB result in data loss? > > Hi Vishal, > > I am not sure I get what you mean by the question #2: > >2. SST files get created each time a checkpoint is triggered. At this > point, does the data for a given key get merged in case the initial data > was read from an SST file while the update must have happened in memory? > > Could you maybe rephrase it? > > Upon the arrival of a checkpoint barrier, Flink flushes all memtables to > SST files synchronously (so any state updates are shortly blocked). > Compaction runs in background thread(s) and Flink deals with the outcome of > RocksDB compaction post factum. For non-incremental checkpoints, everything > is straightforward - all SST files are copied over. The interplay of the > incremental checkpoints and compaction is described in this [1] blog post. > > [1] > https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html > > Best, > Alexander Fedulov > > On Mon, Jul 4, 2022 at 4:25 PM Vishal Surana <vis...@moengage.com> wrote: > > In my load tests, I've found FIFO compaction to offer the best performance > as my job needs state only for so long. However, this particular statement > in RocksDB documentation concerns me: > > "Since we never rewrite the key-value pair, we also don't ever apply the > compaction filter on the keys." > > This is from - > https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style > > I've observed that SST files are getting compacted into larger SST files > until a configured threshold is reached. Thus I'm not sure what's going on > anymore. > > My questions at this stage are: > > 1. If there's a value that I get from RocksDB and I decide to update > this value back then will it work with FIFO compaction? > 2. SST files get created each time a checkpoint is triggered. At > this point, does the data for a given key get merged in case the initial > data was read from an SST file while the update must have happened in > memory? > 3. If the answer to above is yes, then I suppose I can use FIFO > compaction. However, then my question is whether the RocksDB documentation > is wrong or whether Flink is doing something in addition to what RocksDB > does. > > Thanks! > >
