Hi, Junzhi I'm very pleased to see that you and Sicheng Yu are leading the design and development of FastIoTConsensus and IoTV2Consensus. From designing a unified consensus algorithm framework, integrating the strong consistency consensus algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, and now further evolving from operation synchronization to state synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully explored better consensus solutions in time-series scenarios, creating greater value for users with different needs.
I look forward to working together to further refine the consensus layer of IoTDB! Best ----------------- Xinyu Tan On 2024/06/03 10:28:18 Junzhi Peng wrote: > Hello everyone, > > I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu, > the committer of IoTDB, have implemented two new consensus protocols on > DataRegion based on IoTDB Pipe framework: FastIoTConsensus and > IoTV2Consensus, and we are excited to share with you this new feature. > > As we all know, we have been using our self-developed IoTConsensus for > DataRegion in most cases. It has excellent performance and can meet our > business needs well. However, we found that IoTConsensus has availability > issues in some scenarios. For example, after a node goes down, the WAL logs > of other nodes may accumulate quickly, thus blocking writing. In addition, > we also hope to further sacrifice consistency to obtain higher > synchronization performance. Therefore, we introduced new consensus > protocols: FastIoTConsensus and IoTV2Consensus. > > FastIoTConsensus synchronizes the TsFile data file to achieve replica > synchronization, thereby decoupling from WAL and solving the problem of > IoTConsensus replica accumulation. Compared with IoTConsensus, > FastIoTConsensus will further sacrifice consistency and have higher > synchronization latency, but in theory it has higher write throughput > performance and lower storage and computing resource usage. > > IoTV2Consensus also introduces synchronization of TsFile data files to > solve the problem of IoTConsensus WAL accumulation and improve performance. > However, IoTV2Consensus also supports the use of WAL for data > synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the > carrier of data synchronization according to the real-time load of the > replica. IoTV2Consensus uses WAL for data synchronization by default. When > WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data > synchronization. > > At present, we have implemented the above two consensus protocols in this PR > <https://github.com/apache/iotdb/pull/12355>. However, in the future, we > will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we > welcome everyone to use and make suggestions, but the relevant > modifications and iterations will be carried out after FastIoTConsensus is > polished. > > We hope you are interested in this feature and would like to participate in > the development and testing. You can easily modify the > `data_region_consensus_protocol_class` configuration item of current > `iotdb-system.properties` to use the above two new consensus protocols. You > can also leave your comments and suggestions in this thread. Appreciate any > suggestion/feedback & contribution. > > Thank you for your attention and support. > > Best regards, > ————————————— > Junzhi Peng >
