Thanks guys! This was really helpful
On Fri, 21 Jul 2023 at 8:49 AM, tison wrote:
> > This is the case.
>
> Yanfei is correct that the uploading happens during the checkpoint stage.
>
> Before checkpointing, there is a snapshotting stage that synchronously
> takes RocksDB snapshot by forcing all
> This is the case.
Yanfei is correct that the uploading happens during the checkpoint stage.
Before checkpointing, there is a snapshotting stage that synchronously
takes RocksDB snapshot by forcing all memtables into sstables on disk, and
hard-linked in a local temporary directory.
In the check
Hi Taher,
> is flink using rocks-cloud too? Or flink is moving rocks
files after checkpoints?
Flink isn't using rocks-cloud. flink would upload local rocks files to
s3/DFS during checkpointing. Here is the related code:
- RocksNativeFullSnapshotStrategy:
https://github.com/apache/flink/blob/mas
> flink is moving rocks
> files after checkpoints
This is the case.
You can read the code and docs around EmbeddedRocksDBStateBackend and
FileSystemCheckpointStorage.
> Please share
N.B. An imperative tone can read uncomfortable. Maybe it's not your
intention but this read like a command.
Bes
Hi all,
I know that Flink uses rocksdb to write checkpoints. While
researching rocksdb i found that there is something called rocks-cloud that
writes data over to s3. I know flink also does checkpoints over rocksdb to
s3. Question is, is flink using rocks-cloud too? Or flink is moving rock
