[
https://issues.apache.org/jira/browse/FLINK-28699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573649#comment-17573649
]
Yun Tang commented on FLINK-28699:
----------------------------------
I think adding {{NO_SHARING}} flag to RocksIncrementalSnapshotStrategy might
not be a good idea as the current RocksIncrementalSnapshotStrategy could handle
different sharing files strategies according to the passed checkpoint options.
And I believe the code change should not be big as we can share most of the
code for the newly {{RocksNativeFullSnapshotStrategy}} with the existing
{{{}RocksIncrementalSnapshotStrategy{}}}.
> Native rocksdb full snapshot in non-incremental checkpointing
> -------------------------------------------------------------
>
> Key: FLINK-28699
> URL: https://issues.apache.org/jira/browse/FLINK-28699
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.14.5, 1.15.1
> Reporter: Lihe Ma
> Assignee: Lihe Ma
> Priority: Major
> Labels: pull-request-available
>
> When rocksdb statebackend is used and state.backend.incremental enabled,
> flink will figure out newly created sst files generated by rocksdb during
> checkpoint, and read all the states from rocksdb and write to files during
> savepoint [1].
> When state.backend.incremental disabled, flink will read all the states from
> rocksdb and generate state files in checkpoint and savepoint [2]. This makes
> sense in savepoint, cause user can take a savepoint with rocksdb statebackend
> and then restore it using another statebackend, but in checkpoint,
> deserialisation and serialisation of state results in performance loss.
> If the native rocksdb snapshot is introduced in full snapshot, theoretically
> better performance can be achieved. At the same time, savepoint remains the
> same as before.
>
> #
> https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java
> #
> https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksFullSnapshotStrategy.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
