Dear Junzhi Peng and Sicheng Yu, Thank you for sharing the exciting news about the new consensus protocols, FastIoTConsensus and IoTV2Consensus. Your efforts and contributions to improving the IoTDB project are greatly appreciated. We look forward to exploring these new features and providing feedback.
Best regards, Itami Sho 2024年6月3日 18:28，Junzhi Peng <junzhi.peng...@gmail.com> 写道： Hello everyone, I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu, the committer of IoTDB, have implemented two new consensus protocols on DataRegion based on IoTDB Pipe framework: FastIoTConsensus and IoTV2Consensus, and we are excited to share with you this new feature. As we all know, we have been using our self-developed IoTConsensus for DataRegion in most cases. It has excellent performance and can meet our business needs well. However, we found that IoTConsensus has availability issues in some scenarios. For example, after a node goes down, the WAL logs of other nodes may accumulate quickly, thus blocking writing. In addition, we also hope to further sacrifice consistency to obtain higher synchronization performance. Therefore, we introduced new consensus protocols: FastIoTConsensus and IoTV2Consensus. FastIoTConsensus synchronizes the TsFile data file to achieve replica synchronization, thereby decoupling from WAL and solving the problem of IoTConsensus replica accumulation. Compared with IoTConsensus, FastIoTConsensus will further sacrifice consistency and have higher synchronization latency, but in theory it has higher write throughput performance and lower storage and computing resource usage. IoTV2Consensus also introduces synchronization of TsFile data files to solve the problem of IoTConsensus WAL accumulation and improve performance. However, IoTV2Consensus also supports the use of WAL for data synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the carrier of data synchronization according to the real-time load of the replica. IoTV2Consensus uses WAL for data synchronization by default. When WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data synchronization. At present, we have implemented the above two consensus protocols in this PR <https://github.com/apache/iotdb/pull/12355>. However, in the future, we will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we welcome everyone to use and make suggestions, but the relevant modifications and iterations will be carried out after FastIoTConsensus is polished. We hope you are interested in this feature and would like to participate in the development and testing. You can easily modify the `data_region_consensus_protocol_class` configuration item of current `iotdb-system.properties` to use the above two new consensus protocols. You can also leave your comments and suggestions in this thread. Appreciate any suggestion/feedback & contribution. Thank you for your attention and support. Best regards, ————————————— Junzhi Peng
