What's the mode of changing things in IoTDB?
Hi all, So, before I start doing a few small little things (Like adding the SBOM generation etc.), I wanted to double check the mode of operation here. As most discussions are happening in WeChat and Feishu, it’s a bit hard for me to participate (Even if Feischu is a lot better as I can have it auto-translate). Is it ok to commit small changes like adding the Sbom generation to the development banch, or is everything RTC (Review Then Commit … aka do all changes on branches and have them merged by others)? Chris
Re: What's the mode of changing things in IoTDB?
Hi Chris, In general, RTC is the way to go, even if the changes are small. Best, --- Yuan Tian On Tue, Jul 18, 2023 at 3:59 PM Christofer Dutz wrote: > Hi all, > > So, before I start doing a few small little things (Like adding the SBOM > generation etc.), I wanted to double check the mode of operation here. > As most discussions are happening in WeChat and Feishu, it’s a bit hard > for me to participate (Even if Feischu is a lot better as I can have it > auto-translate). > Is it ok to commit small changes like adding the Sbom generation to the > development banch, or is everything RTC (Review Then Commit … aka do all > changes on branches and have them merged by others)? > > Chris >
AW: What's the mode of changing things in IoTDB?
Glad I asked ;-) Chris Von: Yuan Tian Datum: Dienstag, 18. Juli 2023 um 10:25 An: dev@iotdb.apache.org Betreff: Re: What's the mode of changing things in IoTDB? Hi Chris, In general, RTC is the way to go, even if the changes are small. Best, --- Yuan Tian On Tue, Jul 18, 2023 at 3:59 PM Christofer Dutz wrote: > Hi all, > > So, before I start doing a few small little things (Like adding the SBOM > generation etc.), I wanted to double check the mode of operation here. > As most discussions are happening in WeChat and Feishu, it’s a bit hard > for me to participate (Even if Feischu is a lot better as I can have it > auto-translate). > Is it ok to commit small changes like adding the Sbom generation to the > development banch, or is everything RTC (Review Then Commit … aka do all > changes on branches and have them merged by others)? > > Chris >
[PROPOSAL] Enhance Read Consistency Level During Restart in RatisConsensus
Hi dev, I'd like to draw your attention to an existing issue in our current read consistency level within the RatisConsensus module. As it stands, the default level is set to "query statemachine directly”, which, while latency-friendly, has led to user-reported bugs. Specifically, these bugs relate to the production of inconsistent results in subsequent SQL queries during a restart, creating a phantom read problem that may be confusing for our users. To address this issue, I propose that we temporarily increase the read consistency level to linearizable read during restarts. This will ensure that we maintain data consistency during the critical recovery period. Once the cluster has successfully finished recovering from previous logs, we can then revert to the default consistency level. You can find more details about this proposed solution in the linked pull request: https://github.com/apache/iotdb/pull/10597。 **Please note** that this change may affect module (including CQ, schema region, and data region) that calls RatisConsensus.read during the restart process. In such cases, a RatisUnderRecoveryException may be returned, indicating that RatisConsensus cannot serve read requests while it's replaying RaftLog. Therefore, we strongly encourage the affected modules to handle this situation appropriately, such as implementing a retry mechanism. I look forward to hearing your thoughts on this proposal. Your feedback and suggestions will be appreciated. Regards William Song
[BUILD-FAILURE]: Job 'IoTDB/IoTDB-pip-new/master [master] [292]'
BUILD-FAILURE: Job 'IoTDB/IoTDB-pip-new/master [master] [292]': Check console output at "https://ci-builds.apache.org/job/IoTDB/job/IoTDB-pip-new/job/master/292/";>IoTDB/IoTDB-pip-new/master [master] [292]"
AW: [PROPOSAL] Enhance Read Consistency Level During Restart in RatisConsensus
Hi, I agree that it’s better to have the defaults produce safer (more consistent) results and document optimization options for users, that want/need them and know about potential drawbacks. Admittedly I’m not yet too deep in the internals of IoTDB, but at least this would be my expectation on a user-level. I’m currently reviewing our “competitor” solutions and inconsistencies were what made me dislike the one or the other solution instantly. Trust lost easily, and hard to regain. Chris Von: William Song Datum: Mittwoch, 19. Juli 2023 um 04:14 An: dev@iotdb.apache.org Betreff: [PROPOSAL] Enhance Read Consistency Level During Restart in RatisConsensus Hi dev, I'd like to draw your attention to an existing issue in our current read consistency level within the RatisConsensus module. As it stands, the default level is set to "query statemachine directly”, which, while latency-friendly, has led to user-reported bugs. Specifically, these bugs relate to the production of inconsistent results in subsequent SQL queries during a restart, creating a phantom read problem that may be confusing for our users. To address this issue, I propose that we temporarily increase the read consistency level to linearizable read during restarts. This will ensure that we maintain data consistency during the critical recovery period. Once the cluster has successfully finished recovering from previous logs, we can then revert to the default consistency level. You can find more details about this proposed solution in the linked pull request: https://github.com/apache/iotdb/pull/10597。 **Please note** that this change may affect module (including CQ, schema region, and data region) that calls RatisConsensus.read during the restart process. In such cases, a RatisUnderRecoveryException may be returned, indicating that RatisConsensus cannot serve read requests while it's replaying RaftLog. Therefore, we strongly encourage the affected modules to handle this situation appropriately, such as implementing a retry mechanism. I look forward to hearing your thoughts on this proposal. Your feedback and suggestions will be appreciated. Regards William Song
