[PR] [IOTDB-6267] Load 2.0 [iotdb]
yschengzi opened a new pull request, #11705: URL: https://github.com/apache/iotdb/pull/11705 - MPP Load has some problems with stability and compatibility with the Pipe system, and there is room for optimization in loading speed. Issue 1: Uncontrollable upper limit of total memory used when multiple Load statements are executed concurrently. - Currently, Load only strictly controls the upper limit of memory used by a single Load statement during its execution life cycle. - When a large number of Load statements are executed concurrently, the total memory size used by these Load statements is uncontrollable. - Please refer to MPP Load memory footprint for the memory usage during the execution life cycle of a single Load statement. Issue 2: New data added by Load is not properly recognized by the Pipe system. - The Pipe system currently adds a ProgressIndex to all new data added to the IoTDB (see the discussion of Key Issues in Pipe System Design and Implementation). - In a normal write process, the process of adding the index is realized by the consensus layer. - In the normal write process, the process of adding an identifier is implemented by the consensus layer. However, the current Load's two-phase transaction commit process does not go through the consensus layer, and does not have a normal progress identifier, nor can it be correctly recognized by the Pipe system when restarting the task. Issue 3: Too many serialization steps in the Load TsFile process. - In the LoadTsFileScheduler class, the implementation of MPP Load 1.0 is to - Iterate through each TsFile in the TsFile. - For each TsFile, perform split first, then send, and then perform the second stage after all the sends are completed. - After completing the second phase, the next TsFile is loaded sequentially. - Since the TsFile splitting process may involve memory computation, the disk IO capacity is not fully utilized during the memory computation. - A single TsFile is serialized during splitting and sending via Thrift, waiting for both disk IO and network IO. document link:https://apache-iotdb.feishu.cn/docx/UE9Od5caDoLoYJxt4Ptc4s0hnof -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IOTDB-6199] Support automatically fetch available DN list in Session & SessionPool [iotdb]
sonarcloud[bot] commented on PR #11704: URL: https://github.com/apache/iotdb/pull/11704#issuecomment-1853422480 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11704) **Quality Gate failed** Failed conditions [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11704&metric=new_coverage&view=list) (required ≥ 80%) [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11704) (required ≥ A) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11704)  Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [IOTDB-6199] Support automatically fetch available DN list in Session & SessionPool [iotdb]
JackieTien97 opened a new pull request, #11704: URL: https://github.com/apache/iotdb/pull/11704 more details can be seen in https://apache-iotdb.feishu.cn/docx/AVusdVRc6oLiyvxN0VUclQ72n9g -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] change default config value of inner_compaction_task_selection_mods_file_threshold [iotdb]
caozj1011 opened a new pull request, #11703: URL: https://github.com/apache/iotdb/pull/11703 ## Description change default config value of inner_compaction_task_selection_mods_file_threshold from **50M** to **10M** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix snapshot UT [iotdb]
sonarcloud[bot] commented on PR #11697: URL: https://github.com/apache/iotdb/pull/11697#issuecomment-1853379246 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11697) **Quality Gate passed** Kudos, no new issues were introduced! [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&inNewCodePeriod=true) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11697&resolved=false&inNewCodePeriod=true) No data about Coverage [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11697&metric=new_duplicated_lines_density&view=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11697) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Implement Cluster ID [iotdb]
liyuheng5 opened a new pull request, #11702: URL: https://github.com/apache/iotdb/pull/11702 doc: https://apache-iotdb.feishu.cn/docx/E5hjdaWIwo2JT7xCi6bcvtyinfh -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe schema: RPC request & response datastructures [iotdb]
SteveYurongSu merged PR #11680: URL: https://github.com/apache/iotdb/pull/11680 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enable jenkins log when ci failed [iotdb]
sonarcloud[bot] commented on PR #11701: URL: https://github.com/apache/iotdb/pull/11701#issuecomment-1853187952 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11701) **Quality Gate passed** Kudos, no new issues were introduced! [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11701&resolved=false&inNewCodePeriod=true) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11701&resolved=false&inNewCodePeriod=true) No data about Coverage No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11701) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enable Jenkins Log when CI failed [iotdb]
sonarcloud[bot] commented on PR #11700: URL: https://github.com/apache/iotdb/pull/11700#issuecomment-1853184835 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11700) **Quality Gate passed** Kudos, no new issues were introduced! [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11700&resolved=false&inNewCodePeriod=true) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11700&resolved=false&inNewCodePeriod=true) No data about Coverage No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11700) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enable jenkins log when ci failed [iotdb]
HTHou merged PR #11701: URL: https://github.com/apache/iotdb/pull/11701 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Enable jenkins log when ci failed [iotdb]
OneSizeFitsQuorum opened a new pull request, #11701: URL: https://github.com/apache/iotdb/pull/11701 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enable Jenkins Log when CI failed [iotdb]
HTHou merged PR #11700: URL: https://github.com/apache/iotdb/pull/11700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Enable Jenkins Log when CI failed [iotdb]
OneSizeFitsQuorum opened a new pull request, #11700: URL: https://github.com/apache/iotdb/pull/11700 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Introduce node-level lock to PBTree memory manage [iotdb]
MarcosZyk closed pull request #11605: Introduce node-level lock to PBTree memory manage URL: https://github.com/apache/iotdb/pull/11605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Introduce Node-Level lock to PBTree cache management [iotdb]
Cpaulyz merged PR #11633: URL: https://github.com/apache/iotdb/pull/11633 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] PBTree Flush Control Monitor [iotdb]
Cpaulyz commented on code in PR #11699:
URL: https://github.com/apache/iotdb/pull/11699#discussion_r1424777548
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/metric/SchemaEngineCachedMetric.java:
##
@@ -109,15 +100,15 @@ public void bindTo(AbstractMetricService metricService) {
metricService.createAutoGauge(
Metric.SCHEMA_ENGINE.toString(),
MetricLevel.IMPORTANT,
-CacheMemoryManager.getInstance(),
-CacheMemoryManager::getReleaseThreadNum,
+ReleaseFlushMonitor.getInstance(),
+ReleaseFlushMonitor::getActiveWorkerNum,
Tag.NAME.toString(),
RELEASE_THREAD_NUM);
metricService.createAutoGauge(
Metric.SCHEMA_ENGINE.toString(),
MetricLevel.IMPORTANT,
-CacheMemoryManager.getInstance(),
-CacheMemoryManager::getFlushThreadNum,
+ReleaseFlushMonitor.getInstance(),
+ReleaseFlushMonitor::getActiveWorkerNum,
Review Comment:
Merge these two metrics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] PBTree Flush Control Monitor [iotdb]
Cpaulyz opened a new pull request, #11699: URL: https://github.com/apache/iotdb/pull/11699 ## Description ### Content1 ... ### Content2 ... ### Content3 ... This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IOTDB-6266] Add the ability to flush syncIndex and update reader periodically for IoTConsensus [iotdb]
OneSizeFitsQuorum merged PR #11691: URL: https://github.com/apache/iotdb/pull/11691 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] PBTree Improvement: concurrent operation and interleaved flush [iotdb]
bigreybear opened a new pull request, #11698: URL: https://github.com/apache/iotdb/pull/11698 ## Description ### Content1 ... ### Content2 ... ### Content3 ... This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Make slimit and soffset available in align by device with template [iotdb]
JackieTien97 merged PR #11696: URL: https://github.com/apache/iotdb/pull/11696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix snapshot UT [iotdb]
codecov-commenter commented on PR #11697: URL: https://github.com/apache/iotdb/pull/11697#issuecomment-1851665880 ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/11697?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`09b7e86`)](https://app.codecov.io/gh/apache/iotdb/commit/09b7e86fd6341c133d101b63a09b6dd1e89bcf1b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 49.10% compared to head [(`580c995`)](https://app.codecov.io/gh/apache/iotdb/pull/11697?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 49.09%. > Report is 3 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #11697 +/- ## - Coverage 49.10% 49.09% -0.02% - Complexity2498224988 +6 Files 2817 2817 Lines176773 176794 +21 Branches 2118621194 +8 - Hits 8680486792 -12 - Misses8996990002 +33 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/11697?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Show devices with template [iotdb]
Cpaulyz commented on code in PR #11681:
URL: https://github.com/apache/iotdb/pull/11681#discussion_r1423699230
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/header/ColumnHeaderConstant.java:
##
@@ -49,6 +49,8 @@ private ColumnHeaderConstant() {
public static final String DEADBAND = "Deadband";
public static final String DEADBAND_PARAMETERS = "DeadbandParameters";
public static final String IS_ALIGNED = "IsAligned";
+ public static final String Template = "Template";
Review Comment:
Keep the code style consistent with the original code.
```suggestion
public static final String TEMPLATE = "Template";
```
##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/filter/SchemaFilterFactory.java:
##
@@ -44,6 +45,10 @@ public static SchemaFilter createViewTypeFilter(ViewType
viewType) {
return new ViewTypeFilter(viewType);
}
+ public static SchemaFilter createTemplateIdFilter(String templateName,
boolean isEqual) {
Review Comment:
Please let the name of the method express the exact meaning.
```suggestion
public static SchemaFilter createTemplateNameFilter(String templateName,
boolean isEqual) {
```
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/schema/source/DeviceSchemaSource.java:
##
@@ -96,15 +97,28 @@ public void transformToTsBlockColumns(
builder
.getColumnBuilder(0)
.writeBinary(new Binary(device.getFullPath(),
TSFileConfig.STRING_CHARSET));
+
+String TemplateName = "null";
Review Comment:
Do not use MAGIC NUMBER.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Show devices with template [iotdb]
MiniSho commented on code in PR #11681:
URL: https://github.com/apache/iotdb/pull/11681#discussion_r1423697324
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/common/header/ColumnHeaderConstant.java:
##
@@ -49,6 +49,8 @@ private ColumnHeaderConstant() {
public static final String DEADBAND = "Deadband";
public static final String DEADBAND_PARAMETERS = "DeadbandParameters";
public static final String IS_ALIGNED = "IsAligned";
+ public static final String Template = "Template";
Review Comment:
Capitalize `Template` :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] SeriesScanUtil support predicate pushdown [iotdb]
sonarcloud[bot] commented on PR #11671: URL: https://github.com/apache/iotdb/pull/11671#issuecomment-1851602441 SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11671) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11671&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11671&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11671&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11671&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11671&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11671&metric=new_coverage&view=list) [](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11671&metric=new_duplicated_lines_density&view=list) [0.7% Duplication](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11671&metric=new_duplicated_lines_density&view=list)  Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IOTDB-6266] Add the ability to flush syncIndex and update reader periodically for IoTConsensus [iotdb]
sonarcloud[bot] commented on PR #11691: URL: https://github.com/apache/iotdb/pull/11691#issuecomment-1851587390 SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11691) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11691&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11691&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11691&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11691&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11691&metric=new_coverage&view=list) [0.0% Coverage](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11691&metric=new_coverage&view=list) [](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11691&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11691&metric=new_duplicated_lines_density&view=list)  Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=sonarcloud-welcome) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix snapshot UT [iotdb]
sonarcloud[bot] commented on PR #11697: URL: https://github.com/apache/iotdb/pull/11697#issuecomment-1851580165 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=11697) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11697&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11697&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=11697&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=11697&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11697&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11697&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=11697&metric=new_duplicated_lines_density&view=list) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix snapshot UT [iotdb]
THUMarkLau opened a new pull request, #11697: URL: https://github.com/apache/iotdb/pull/11697 Fix IoTDBSnapshotTest -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IOTDB-6266] Add the ability to flush syncIndex and update reader periodically for IoTConsensus [iotdb]
OneSizeFitsQuorum commented on code in PR #11691:
URL: https://github.com/apache/iotdb/pull/11691#discussion_r1423665216
##
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/logdispatcher/IndexController.java:
##
@@ -101,12 +102,15 @@ public long getLastFlushedIndex() {
}
private void checkPersist(boolean forcePersist) {
-if (forcePersist || currentIndex - lastFlushedIndex >= checkpointGap) {
+if (forcePersist
+|| currentIndex - lastFlushedIndex >= checkpointGap
+|| System.nanoTime() - updateTime >=
IoTConsensus.READER_UPDATE_INTERVAL_IN_NS) {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [IOTDB-6266] Add the ability to flush syncIndex and update reader periodically for IoTConsensus [iotdb]
OneSizeFitsQuorum commented on code in PR #11691:
URL: https://github.com/apache/iotdb/pull/11691#discussion_r1423662523
##
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/logdispatcher/LogDispatcher.java:
##
@@ -157,6 +159,15 @@ public synchronized OptionalLong getMinFlushedSyncIndex() {
return
threads.stream().mapToLong(LogDispatcherThread::getLastFlushedSyncIndex).min();
}
+ public void checkAndFlushIndex() {
+threads.forEach(
+thread -> {
+ IndexController controller = thread.getController();
+ controller.update(controller.getCurrentIndex(), false);
Review Comment:
Good suggestion！
##
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/logdispatcher/LogDispatcher.java:
##
@@ -157,6 +159,15 @@ public synchronized OptionalLong getMinFlushedSyncIndex() {
return
threads.stream().mapToLong(LogDispatcherThread::getLastFlushedSyncIndex).min();
}
+ public void checkAndFlushIndex() {
+threads.forEach(
+thread -> {
+ IndexController controller = thread.getController();
+ controller.update(controller.getCurrentIndex(), false);
Review Comment:
Good suggestion！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Optimized wal file deletion algorithm [iotdb]
HeimingZ commented on code in PR #11682:
URL: https://github.com/apache/iotdb/pull/11682#discussion_r1423610935
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##
@@ -583,22 +578,28 @@ private void snapshotMemTable(DataRegion dataRegion, File
tsFile, MemTableInfo m
}
}
-public long initFirstValidWALVersionId() {
- long firstVersionId = checkpointManager.getFirstValidWALVersionId();
- // This means that the relevant memTable in the file has been
successfully flushed, so we
- // should scroll through a new wal file so that the current file can be
deleted
- if (firstVersionId == Long.MIN_VALUE) {
-// roll wal log writer to delete current wal file
-if (buffer.getCurrentWALFileSize() > 0) {
- rollWALFile();
-}
-// update firstValidVersionId
-firstVersionId = checkpointManager.getFirstValidWALVersionId();
-if (firstVersionId == Long.MIN_VALUE) {
- firstVersionId = buffer.getCurrentWALFileVersion();
+public boolean isContainsActiveOrPinnedMemTable(Long versionId) {
+ Set memTableIdsOfCurrentWal = memTableIdsOfWalMap.get(versionId);
+ // If this set is empty, there is a case where WalEntry has been logged
but not persisted,
+ // because WalEntry is persisted asynchronously. In this case, the file
cannot be deleted
+ // directly, so it is considered active
+ if (memTableIdsOfCurrentWal.isEmpty()) {
+return true;
+ }
+ for (MemTableInfo memTableInfo : activeOrPinnedMemTables) {
+for (Long memTableId : memTableIdsOfCurrentWal) {
+ if (memTableId.equals(memTableInfo.getMemTableId())) {
+return true;
+ }
}
}
Review Comment:
May we use Collections.disjoint() here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Optimized wal file deletion algorithm [iotdb]
HeimingZ commented on code in PR #11682:
URL: https://github.com/apache/iotdb/pull/11682#discussion_r1423590781
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##
@@ -252,20 +254,24 @@ public DeleteOutdatedFileTask() {
}
private void init() {
- this.firstValidVersionId = initFirstValidWALVersionId();
Review Comment:
We don‘t need firstValidWALVersionId any more, but I think we should keep
initFirstValidWALVersionId method because the delete procedure will try to
delete current file in this method.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##
@@ -252,20 +254,24 @@ public DeleteOutdatedFileTask() {
}
private void init() {
- this.firstValidVersionId = initFirstValidWALVersionId();
- this.filesShouldDelete =
logDirectory.listFiles(this::filterFilesToDelete);
- if (filesShouldDelete == null) {
-filesShouldDelete = new File[0];
+ File[] allWalFilesOfOneNode = WALFileUtils.listAllWALFiles(logDirectory);
+ if (allWalFilesOfOneNode == null) {
+allWalFilesOfOneNode = new File[0];
}
+ WALFileUtils.ascSortByVersionId(allWalFilesOfOneNode);
+ this.sortedWalFilesExcludingLast =
+ Arrays.copyOfRange(allWalFilesOfOneNode, 0,
allWalFilesOfOneNode.length - 1);
Review Comment:
Exception occurs when allWalFilesOfOneNode.length==0
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##
@@ -252,20 +254,24 @@ public DeleteOutdatedFileTask() {
}
private void init() {
- this.firstValidVersionId = initFirstValidWALVersionId();
Review Comment:
We don‘t need firstValidWALVersionId any more, but I think we should keep
initFirstValidWALVersionId method because the delete procedure will try to
delete current file in this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [IOTDB-6266] Add the ability to flush syncIndex and update reader periodically for IoTConsensus [iotdb]
BUAAserein commented on code in PR #11691:
URL: https://github.com/apache/iotdb/pull/11691#discussion_r1422243185
##
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/logdispatcher/LogDispatcher.java:
##
@@ -157,6 +159,15 @@ public synchronized OptionalLong getMinFlushedSyncIndex() {
return
threads.stream().mapToLong(LogDispatcherThread::getLastFlushedSyncIndex).min();
}
+ public void checkAndFlushIndex() {
+threads.forEach(
+thread -> {
+ IndexController controller = thread.getController();
+ controller.update(controller.getCurrentIndex(), false);
Review Comment:
maybe we can use `controller.update(controller.getCurrentIndex(), true);` to
force update periodically
##
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/logdispatcher/IndexController.java:
##
@@ -101,12 +102,15 @@ public long getLastFlushedIndex() {
}
private void checkPersist(boolean forcePersist) {
-if (forcePersist || currentIndex - lastFlushedIndex >= checkpointGap) {
+if (forcePersist
+|| currentIndex - lastFlushedIndex >= checkpointGap
+|| System.nanoTime() - updateTime >=
IoTConsensus.READER_UPDATE_INTERVAL_IN_NS) {
Review Comment:
in this way, we don't need to do the time check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
