[Proposal] New Consensus Protocals for IoTDB DataRegion

[Junzhi Peng]

Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xiangpeng Hu]

Hello Junzhi,


I would like to extend my heartfelt thanks to you and Sicheng Yu for your 
FastIoTConsensus and IoTV2Consensus protocols. It is a significant step forward 
for IoTDB.


I am well aware that the existing IoTConsensus has been performing 
exceptionally well in most scenarios, yet it has encountered availability 
issues. I appreciate your insight into the need to further sacrifice 
consistency for higher synchronization performance, which is a forward-thinking 
approach. I am also very excited about the opportunity to participate in the 
development and testing of these new consensus protocols.


Lastly, your hard work and innovation are greatly appreciated, and I look 
forward to seeing the real-world performance of FastIoTConsensus and 
IoTV2Consensus.


Best regards,
Xiangpeng Hu
 Replied Message 
| From | Junzhi Peng |
| Date | 6/3/2024 18:28 |
| To |  |
| Subject | [Proposal] New Consensus Protocals for IoTDB DataRegion |
Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
<https://github.com/apache/iotdb/pull/12355>. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xinyu Tan]

Hi, Junzhi

I'm very pleased to see that you and Sicheng Yu are leading the design and 
development of FastIoTConsensus and IoTV2Consensus. From designing a unified 
consensus algorithm framework, integrating the strong consistency consensus 
algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, 
and now further evolving from operation synchronization to state 
synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully 
explored better consensus solutions in time-series scenarios, creating greater 
value for users with different needs.

I look forward to working together to further refine the consensus layer of 
IoTDB!

Best
-
Xinyu Tan

On 2024/06/03 10:28:18 Junzhi Peng wrote:
> Hello everyone,
> 
> I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
> the committer of IoTDB, have implemented two new consensus protocols on
> DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
> IoTV2Consensus, and we are excited to share with you this new feature.
> 
> As we all know, we have been using our self-developed IoTConsensus for
> DataRegion in most cases. It has excellent performance and can meet our
> business needs well. However, we found that IoTConsensus has availability
> issues in some scenarios. For example, after a node goes down, the WAL logs
> of other nodes may accumulate quickly, thus blocking writing. In addition,
> we also hope to further sacrifice consistency to obtain higher
> synchronization performance. Therefore, we introduced new consensus
> protocols: FastIoTConsensus and IoTV2Consensus.
> 
> FastIoTConsensus synchronizes the TsFile data file to achieve replica
> synchronization, thereby decoupling from WAL and solving the problem of
> IoTConsensus replica accumulation. Compared with IoTConsensus,
> FastIoTConsensus will further sacrifice consistency and have higher
> synchronization latency, but in theory it has higher write throughput
> performance and lower storage and computing resource usage.
> 
> IoTV2Consensus also introduces synchronization of TsFile data files to
> solve the problem of IoTConsensus WAL accumulation and improve performance.
> However, IoTV2Consensus also supports the use of WAL for data
> synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
> carrier of data synchronization according to the real-time load of the
> replica. IoTV2Consensus uses WAL for data synchronization by default. When
> WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
> synchronization.
> 
> At present, we have implemented the above two consensus protocols in this PR
> . However, in the future, we
> will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
> welcome everyone to use and make suggestions, but the relevant
> modifications and iterations will be carried out after FastIoTConsensus is
> polished.
> 
> We hope you are interested in this feature and would like to participate in
> the development and testing. You can easily modify the
> `data_region_consensus_protocol_class` configuration item of current
> `iotdb-system.properties` to use the above two new consensus protocols. You
> can also leave your comments and suggestions in this thread. Appreciate any
> suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> —
> Junzhi Peng
> 

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xinyu Tan]

Hi, Junzhi

I'm very pleased to see that you and Sicheng Yu are leading the design and 
development of FastIoTConsensus and IoTV2Consensus. From designing a unified 
consensus algorithm framework, integrating the strong consistency consensus 
algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, 
and now further evolving from operation synchronization to state 
synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully 
explored better consensus solutions in time-series scenarios, creating greater 
value for users with different needs.

I look forward to working together to further refine the consensus layer of 
IoTDB!

Best
-
Xinyu Tan

On 2024/06/03 10:28:18 Junzhi Peng wrote:
> Hello everyone,
> 
> I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
> the committer of IoTDB, have implemented two new consensus protocols on
> DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
> IoTV2Consensus, and we are excited to share with you this new feature.
> 
> As we all know, we have been using our self-developed IoTConsensus for
> DataRegion in most cases. It has excellent performance and can meet our
> business needs well. However, we found that IoTConsensus has availability
> issues in some scenarios. For example, after a node goes down, the WAL logs
> of other nodes may accumulate quickly, thus blocking writing. In addition,
> we also hope to further sacrifice consistency to obtain higher
> synchronization performance. Therefore, we introduced new consensus
> protocols: FastIoTConsensus and IoTV2Consensus.
> 
> FastIoTConsensus synchronizes the TsFile data file to achieve replica
> synchronization, thereby decoupling from WAL and solving the problem of
> IoTConsensus replica accumulation. Compared with IoTConsensus,
> FastIoTConsensus will further sacrifice consistency and have higher
> synchronization latency, but in theory it has higher write throughput
> performance and lower storage and computing resource usage.
> 
> IoTV2Consensus also introduces synchronization of TsFile data files to
> solve the problem of IoTConsensus WAL accumulation and improve performance.
> However, IoTV2Consensus also supports the use of WAL for data
> synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
> carrier of data synchronization according to the real-time load of the
> replica. IoTV2Consensus uses WAL for data synchronization by default. When
> WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
> synchronization.
> 
> At present, we have implemented the above two consensus protocols in this PR
> . However, in the future, we
> will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
> welcome everyone to use and make suggestions, but the relevant
> modifications and iterations will be carried out after FastIoTConsensus is
> polished.
> 
> We hope you are interested in this feature and would like to participate in
> the development and testing. You can easily modify the
> `data_region_consensus_protocol_class` configuration item of current
> `iotdb-system.properties` to use the above two new consensus protocols. You
> can also leave your comments and suggestions in this thread. Appreciate any
> suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> —
> Junzhi Peng
> 

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[ITAMI SHO]

Dear Junzhi Peng and Sicheng Yu,

Thank you for sharing the exciting news about the new consensus protocols, 
FastIoTConsensus and IoTV2Consensus. Your efforts and contributions to 
improving the IoTDB project are greatly appreciated. We look forward to 
exploring these new features and providing feedback.

Best regards,

Itami Sho

2024年6月3日 18:28，Junzhi Peng  写道：

Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng