zoltar9264 commented on code in PR #22669:
URL: https://github.com/apache/flink/pull/22669#discussion_r1211024303
##########
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksDBSnapshotStrategyBase.java:
##########
@@ -395,18 +394,16 @@ public void release() {
/** Previous snapshot with uploaded sst files. */
protected static class PreviousSnapshot {
- @Nullable private final Map<StateHandleID, Long> confirmedSstFiles;
+ @Nullable private final Map<StateHandleID, StreamStateHandle>
confirmedSstFiles;
- protected PreviousSnapshot(@Nullable Map<StateHandleID, Long>
confirmedSstFiles) {
+ protected PreviousSnapshot(
+ @Nullable Map<StateHandleID, StreamStateHandle>
confirmedSstFiles) {
this.confirmedSstFiles = confirmedSstFiles;
}
protected Optional<StreamStateHandle> getUploaded(StateHandleID
stateHandleID) {
if (confirmedSstFiles != null &&
confirmedSstFiles.containsKey(stateHandleID)) {
- // we introduce a placeholder state handle, that is replaced
with the
- // original from the shared state registry (created from a
previous checkpoint)
- return Optional.of(
- new
PlaceholderStreamStateHandle(confirmedSstFiles.get(stateHandleID)));
Review Comment:
ByteStreamStateHandle tends to occupy little memory (no more than 20KB by
default), and these memory occupations only occur during network transmission.
When repeated ByteStreamStateHandle is passed to SharedStateRegistry, it will
be discarded. And, I think we should try to avoid the generation of small files
(maybe through file aggregation, etc.).
Using PlaceholderStreamStateHandle makes it difficult to implement
SharedStateRegisterKey based on physical id, although it can be achieved by
storing physical id of original StateHandle into PlaceholderStreamStateHandle.
But I still think removing the PlaceholderStreamStateHandle would make the
SharedStateRegistry much simpler and clearer. Since I haven't observed
bottlenecks in ByteStreamStateHandle delivery in job with 2.6TB of state, I
think the rpc overhead of removing PlaceholderStreamStateHandle is acceptable.
Is the above explanation acceptable to you @rkhachatryan ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
