Jiayi-Liao edited a comment on pull request #8751: URL: https://github.com/apache/flink/pull/8751#issuecomment-848429735
Hi @StephanEwen @carp84 , our team also implements the feature at [Bytedance](https://www.bytedance.com/en/) (Tiktok is one of our popular products)recently, but in a different way without complex changes on JobMaster's side. The core idea is, **Snapshot:** * Use `RocksDBIncrementalPackageStrategy` to package the incremental files into several `SegmentFile` logically (we implement a default strategy based on the `dfs.blocksize` config on Hdfs) * For each `SegmentFile`, write the incremental files to a single file on Hdfs and return a `SegmentFileStateHandle` which also maintains the incremental files' state handles inside * `RocksIncrementalSnapshotStrategy` maintains a mapping between sstable file and `SegmentFile`, to build the `PlaceholderStreamStateHandle` in `IncrementalKeyedStateHandle` * Shared state management use `SegmentFileStateHandle` instead of `StreamStateHandle` to manage the states **Restore:** * Download all the sstable files with the metadata stored in `SegmentFileStateHandle` * Restore the mapping between sstable file and `SegmentFile` in `RocksIncrementalSnapshotStrategy` The good thing is that, we keep the mapping between sstable file and `SegmentFile` on TaskManager side instead of JobMaster side, and there is no change on the shared state management. About the FLIP-158, IMO it can help relieve the impact of small file problem but may not solve it. The number of sstable files can be too many even the write buffer is full every time when being flushed, for example it's normal in our scenarios that several GB incremental state are produced with 20+ sstable files per task, especially in cases like feeding samples to online training models with multiple-join in Flink. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
