Hi Hangxiang, The design looks good, and I also support leaving space for proposal 1.
As you know, loading index/filter/data blocks for querying across levels would introduce high IO access within the LSM tree for old data. If we need to introduce a JNI call during each filesystem call, that would be N times JNI cost compared with the current RocksDB state-backend's JNI cost. The other question is that the configuration of `state.backend.forSt.working-dir` looks too coupled with the ForSt state-backend, how would it be if we introduce another disaggregated state storage? Thus, I think `state.backend.disaggregated.working-dir` might be a better configuration name. Best Yun Tang ________________________________ From: Hangxiang Yu <master...@gmail.com> Sent: Wednesday, March 20, 2024 11:32 To: dev@flink.apache.org <dev@flink.apache.org> Subject: Re: [DISCUSS] FLIP-427: Disaggregated State Store Hi, Yue. Thanks for the reply. If we use proposal1, we can easily reuse these optimizations .It is even > possible to discuss and review the solution together in the Rocksdb > community. We also saw these useful optimizations which could be applied to ForSt in the future. But IIUC, it's not binding to proposal 1, right? We could also implement interfaces about temperature and secondary cache to reuse them, or organize a more complex HybridEnv based on proposal 2. My point is whether we should retain the potential of proposal 1 in the > design. > This is a good suggestion. We choose proposal 2 firstly due to its maintainability and scalability, especially because it could leverage all filesystems flink supported conveniently. Given the indelible advantage in performance, I think we could also consider proposal 1 as an optimization in the future. For the interface on the DB side, we could also expose more different Envs in the future. On Tue, Mar 19, 2024 at 9:14 PM yue ma <mayuefi...@gmail.com> wrote: > Hi Hangxiang, > > Thanks for bringing this discussion. > I have a few questions about the Proposal you mentioned in the FLIP. > > The current conclusion is to use proposal 2, which is okay for me. My point > is whether we should retain the potential of proposal 1 in the design. > There are the following reasons: > 1. No JNI overhead, just like the Performance Part mentioned in Flip > 2. RocksDB currently also provides an interface for Env, and there are also > some implementations, such as HDFS-ENV, which seem to be easily scalable. > 3. The RocksDB community continues to support LSM for different storage > media, such as Tiered Storage > < > https://github.com/facebook/rocksdb/wiki/Tiered-Storage-%28Experimental%29 > > > And some optimizations have been made for this scenario, such as Per > Key Placement Comparison > <https://rocksdb.org/blog/2022/11/09/time-aware-tiered-storage.html>. > *Secondary cache > < > https://github.com/facebook/rocksdb/wiki/SecondaryCache-%28Experimental%29 > >*, > similar to the Hybrid Block Cache mentioned in Flip-423 > If we use proposal1, we can easily reuse these optimizations .It is even > possible to discuss and review the solution together in the Rocksdb > community. > In fact, we have already implemented some production practices using > Proposal1 internally. We have integrated HybridEnv, Tiered Storage, and > Secondary Cache on RocksDB and optimized the performance of Checkpoint and > State Restore. It seems work well for us. > > -- > Best, > Yue > -- Best, Hangxiang.
