Re: [I] [Bug] NPE when deleting timeseries via Java API [iotdb]
bossenti commented on issue #12473: URL: https://github.com/apache/iotdb/issues/12473#issuecomment-2097497917 My bad Please apologize, thank you for your help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NPE when deleting timeseries via Java API [iotdb]
bossenti closed issue #12473: [Bug] NPE when deleting timeseries via Java API URL: https://github.com/apache/iotdb/issues/12473 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Normalize LoadCache interfaces [iotdb]
sonarcloud[bot] commented on PR #12441: URL: https://github.com/apache/iotdb/pull/12441#issuecomment-2097411662 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12441) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12441=new_coverage=list) (required ≥ 80%)  [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12441) (required ≥ A) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12441) ##  Catch issues before they fail your Quality Gate with our IDE extension  [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] iotdb支持ambari组件 [iotdb]
github-actions[bot] commented on issue #12475: URL: https://github.com/apache/iotdb/issues/12475#issuecomment-2097361070 Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] iotdb支持ambari组件 [iotdb]
smileyboy2019 opened a new issue, #12475: URL: https://github.com/apache/iotdb/issues/12475 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar. ### Motivation iotdb 是否可以支持ambari插件，是否可以匹配插件服务 ### Solution _No response_ ### Alternatives _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix region migration code [iotdb]
OneSizeFitsQuorum merged PR #12472: URL: https://github.com/apache/iotdb/pull/12472 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Limit direct buffer memory cost when create consensus region [iotdb]
OneSizeFitsQuorum merged PR #12431: URL: https://github.com/apache/iotdb/pull/12431 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Load: Avoid NPE when load task ends after clean task's execution [iotdb]
SteveYurongSu merged PR #12474: URL: https://github.com/apache/iotdb/pull/12474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NPE when deleting timeseries via Java API [iotdb]
HTHou commented on issue #12473: URL: https://github.com/apache/iotdb/issues/12473#issuecomment-2097237807 Hi @bossenti , `executeQueryStatement` is for query statement only. You should use `executeNonQueryStatement` for delete statement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implement new plan optimizer: AggregationPushDown [iotdb]
sonarcloud[bot] commented on PR #12398: URL: https://github.com/apache/iotdb/pull/12398#issuecomment-2096426862 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12398) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12398=new_coverage=list) (required ≥ 80%)  [7.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12398=new_duplicated_lines_density=list) (required ≤ 3%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12398) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implement new plan optimizer: AggregationPushDown [iotdb]
sonarcloud[bot] commented on PR #12398: URL: https://github.com/apache/iotdb/pull/12398#issuecomment-2096245511 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12398) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12398=new_coverage=list) (required ≥ 80%)  [7.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12398=new_duplicated_lines_density=list) (required ≤ 3%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12398) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Load: Avoid NPE when load task ends after clean task's execution [iotdb]
codecov[bot] commented on PR #12474: URL: https://github.com/apache/iotdb/pull/12474#issuecomment-2096072565 ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/12474?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `0%` with `3 lines` in your changes are missing coverage. Please review. > Project coverage is 42.90%. Comparing base [(`f1b42de`)](https://app.codecov.io/gh/apache/iotdb/commit/f1b42deb3873141bd821bde6129f10cdc3ab14a8?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`fd81603`)](https://app.codecov.io/gh/apache/iotdb/pull/12474?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 5 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/iotdb/pull/12474?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [.../queryengine/execution/load/LoadTsFileManager.java](https://app.codecov.io/gh/apache/iotdb/pull/12474?src=pr=tree=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Fload%2FLoadTsFileManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL2xvYWQvTG9hZFRzRmlsZU1hbmFnZXIuamF2YQ==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/12474?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12474 +/- ## - Coverage 42.93% 42.90% -0.04% Complexity 69 69 Files 3102 3102 Lines191839 191901 +62 Branches 2287022877 +7 - Hits 8237482331 -43 - Misses 109465 109570 +105 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/12474?dropdown=coverage=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Load: Avoid NPE when load task ends after clean task's execution [iotdb]
sonarcloud[bot] commented on PR #12474: URL: https://github.com/apache/iotdb/pull/12474#issuecomment-2095992260 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12474) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb=12474=false=true)  [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_iotdb=12474=new_accepted_issues=list) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb=12474=false=true)  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12474=new_coverage=list)  [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12474=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12474) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] NPE when deleting timeseries via Java API [iotdb]
github-actions[bot] commented on issue #12473: URL: https://github.com/apache/iotdb/issues/12473#issuecomment-2095975803 Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] NPE when deleting timeseries via Java API [iotdb]
bossenti opened a new issue, #12473: URL: https://github.com/apache/iotdb/issues/12473 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar. ### Version 1.3.0 (Java API) ### Describe the bug and provide the minimal reproduce step Deleting a time series via the Java API yields to a NullPointerExecption. ```Java sessionPool.executeQueryStatement(DELETE timeseries root.*) ``` leads to: ```text at org.apache.iotdb.rpc.IoTDBRpcDataSet.(IoTDBRpcDataSet.java:104) at org.apache.iotdb.isession.SessionDataSet.(SessionDataSet.java:121) at org.apache.iotdb.session.SessionConnection.executeQueryStatement(SessionConnection.java:399) at org.apache.iotdb.session.Session.executeStatementMayRedirect(Session.java:791) at org.apache.iotdb.session.Session.executeQueryStatement(Session.java:763) ``` Nevertheless, the time series is deleted so the query is executed successfully but there seems to be an issue when creating the response object. ### What did you expect to see? Executing the query without an exception ### What did you see instead? The stack trace as shown above. ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
JackieTien97 commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1590950784
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/LastQueryOperator.java:
##
@@ -173,4 +178,15 @@ public long calculateRetainedSizeAfterCallingNext() {
}
return max;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
Review Comment:
You should think about the memory size of `tsBlockBuilder` field.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/UpdateLastCacheOperator.java:
##
@@ -89,4 +94,14 @@ protected void appendLastValueToTsBlockBuilder(long
lastTime, TsPrimitiveType la
LastQueryUtil.appendLastValue(
tsBlockBuilder, lastTime, fullPath.getFullPath(),
lastValue.getStringValue(), dataType);
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
+return INSTANCE_SIZE
Review Comment:
where did you add the class size of `MeasurementPath` since you only add
`EstimatedSizeOfPartialPathWithoutClassSize` here.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/OperatorTreeGenerator.java:
##
@@ -2947,6 +2949,18 @@ public List
dealWithConsumeAllChildrenPipelineBreaker(
for (PlanNode localChild : node.getChildren()) {
Operator childOperation = localChild.accept(this, context);
parentPipelineChildren.add(childOperation);
+// if we don't create extra pipeline, the root of the child pipeline
should be current root
+// for example, we have IdentitySinkNode -> DeviceViewNode ->
[ScanNode, ScanNode, ScanNode]
+// the parent of the pipeline of ScanNode should be IdentitySinkNode
in the map, otherwise
+// we will lose the information of these pipelines
+List childPipelineMemoryEstimators =
+
context.getParentPlanNodeIdToMemoryEstimator().get(localChild.getPlanNodeId());
+if (childPipelineMemoryEstimators != null) {
+
context.getParentPlanNodeIdToMemoryEstimator().remove(localChild.getPlanNodeId());
+ context
+ .getParentPlanNodeIdToMemoryEstimator()
+ .put(node.getPlanNodeId(), childPipelineMemoryEstimators);
+}
Review Comment:
What does this mean, and the case is incorrect, because `DeviceViewNode`
won't appear in `dealWithConsumeAllChildrenPipelineBreaker`.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/LastCacheScanOperator.java:
##
@@ -83,4 +87,11 @@ public long calculateRetainedSizeAfterCallingNext() {
public PlanNodeId getSourceId() {
return sourceId;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
+return INSTANCE_SIZE
Review Comment:
also think about the memory usage of `tsBlock` field.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/LastQueryTransformOperator.java:
##
@@ -117,4 +122,13 @@ public void close() throws Exception {
}
tsBlockBuilder = null;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
Review Comment:
also think about `tsBlockBuilder` field's memory size.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/MemoryMeasurable.java:
##
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution;
+
+public interface MemoryMeasurable {
Review Comment:
what's the difference with `Accountable` interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Subscription: randomly generate consumer group id and consumer id if not exist & validate topic config ahead on DN [iotdb]
SteveYurongSu merged PR #12467: URL: https://github.com/apache/iotdb/pull/12467 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe IT: Return directly when SQL retry failed in IT to avoid potential fake failure interference [iotdb]
SteveYurongSu merged PR #12468: URL: https://github.com/apache/iotdb/pull/12468 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Fixed connector subtask manager to allow restarted DataRegions to reuse connectors from other not restarted DataRegions [iotdb]
SteveYurongSu merged PR #12470: URL: https://github.com/apache/iotdb/pull/12470 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Load: Add WritePointRateLimiter to control load writing point rate [iotdb]
Caideyipi commented on code in PR #12356:
URL: https://github.com/apache/iotdb/pull/12356#discussion_r1590868276
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/load/LoadTsFileRateLimiter.java:
##
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.load;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+
+import com.google.common.util.concurrent.RateLimiter;
+
+public class LoadTsFileRateLimiter {
+
+ private static final IoTDBConfig CONFIG =
IoTDBDescriptor.getInstance().getConfig();
+
+ private RateLimiter loadWriteRateLimiter;
+
+ private double throughoutMbPerSec =
CONFIG.getLoadWriteThroughputMbPerSecond();
+
+ public LoadTsFileRateLimiter() {
+loadWriteRateLimiter = RateLimiter.create(throughoutMbPerSec);
+ }
+
+ private void setWritePointRate(final double throughoutMbPerSec) {
+double throughout = throughoutMbPerSec * 1024.0 * 1024.0;
+// if throughout = 0, disable rate limiting
+if (throughout <= 0) {
+ throughout = Double.MAX_VALUE;
Review Comment:
And maybe this means "throughput" too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Load: Add WritePointRateLimiter to control load writing point rate [iotdb]
Caideyipi commented on code in PR #12356:
URL: https://github.com/apache/iotdb/pull/12356#discussion_r1590841482
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/load/LoadTsFileRateLimiter.java:
##
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.load;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+
+import com.google.common.util.concurrent.RateLimiter;
+
+public class LoadTsFileRateLimiter {
+
+ private static final IoTDBConfig CONFIG =
IoTDBDescriptor.getInstance().getConfig();
+
+ private RateLimiter loadWriteRateLimiter;
+
+ private double throughoutMbPerSec =
CONFIG.getLoadWriteThroughputMbPerSecond();
Review Comment:
Does it mean "throughput"?
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/load/LoadTsFileRateLimiter.java:
##
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.load;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+
+import com.google.common.util.concurrent.RateLimiter;
+
+public class LoadTsFileRateLimiter {
+
+ private static final IoTDBConfig CONFIG =
IoTDBDescriptor.getInstance().getConfig();
+
+ private RateLimiter loadWriteRateLimiter;
+
+ private double throughoutMbPerSec =
CONFIG.getLoadWriteThroughputMbPerSecond();
+
+ public LoadTsFileRateLimiter() {
+loadWriteRateLimiter = RateLimiter.create(throughoutMbPerSec);
+ }
+
+ private void setWritePointRate(final double throughoutMbPerSec) {
+double throughout = throughoutMbPerSec * 1024.0 * 1024.0;
+// if throughout = 0, disable rate limiting
+if (throughout <= 0) {
+ throughout = Double.MAX_VALUE;
Review Comment:
..
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/scheduler/load/LoadTsFileScheduler.java:
##
@@ -141,6 +143,7 @@ public LoadTsFileScheduler(
this.allReplicaSets = new HashSet<>();
this.isGeneratedByPipe = isGeneratedByPipe;
this.block =
LoadTsFileMemoryManager.getInstance().allocateDataCacheMemoryBlock();
+this.loadTsFileRateLimiter = new LoadTsFileRateLimiter();
Review Comment:
Make the limiter private per scheduler may not be of use because the
scheduler is created per statement. It is maybe ideal to make a public static
one in loadTsFileManager and use that.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/load/LoadTsFileRateLimiter.java:
##
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ *
Re: [PR] Limit direct buffer memory cost when create consensus region [iotdb]
OneSizeFitsQuorum commented on code in PR #12431:
URL: https://github.com/apache/iotdb/pull/12431#discussion_r1590800057
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java:
##
@@ -3456,12 +3460,37 @@ public void markDeleted() {
writeLock("markDeleted");
try {
deleted = true;
+ releaseDirectBufferMemory();
deletedCondition.signalAll();
} finally {
writeUnlock();
}
}
+ private void acquireDirectBufferMemory() throws DataRegionException {
+long acquireDirectBufferMemCost = 0;
+if
(config.getDataRegionConsensusProtocolClass().equals(ConsensusFactory.IOT_CONSENSUS))
{
+ acquireDirectBufferMemCost = config.getWalBufferSize();
+} else if (config
+.getDataRegionConsensusProtocolClass()
+.equals(ConsensusFactory.RATIS_CONSENSUS)) {
+ acquireDirectBufferMemCost =
config.getDataRatisConsensusLogAppenderBufferSizeMax();
+}
+if
(!SystemInfo.getInstance().addDirectBufferMemoryCost(acquireDirectBufferMemCost))
{
+ throw new DataRegionException(
+ "Total allocated memory for direct buffer will be "
+ + (SystemInfo.getInstance().getDirectBufferMemoryCost() +
acquireDirectBufferMemCost)
+ + ", which is greater than limit mem cost: "
+ +
SystemInfo.getInstance().getTotalDirectBufferMemorySizeLimit());
+}
+this.directBufferMemoryCost = acquireDirectBufferMemCost;
+ }
+
+ private void releaseDirectBufferMemory() {
+
SystemInfo.getInstance().decreaseDirectBufferMemoryCost(directBufferMemoryCost);
+this.directBufferMemoryCost = 0;
Review Comment:
add some comments for repeat deletion
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java:
##
@@ -419,6 +419,24 @@ public void loadProperties(Properties properties) throws
BadNodeUrlException, IO
"max_waiting_time_when_insert_blocked",
Integer.toString(conf.getMaxWaitingTimeWhenInsertBlocked();
+String offHeapMemoryStr = System.getProperty("OFF_HEAP_MEMORY");
+if (offHeapMemoryStr != null) {
+ offHeapMemoryStr = offHeapMemoryStr.toLowerCase();
Review Comment:
abstract a function
##
iotdb-core/datanode/src/assembly/resources/sbin/start-datanode.sh:
##
@@ -163,6 +163,7 @@ launch_service()
iotdb_parms="$iotdb_parms -DTSFILE_CONF=${IOTDB_CONF}"
iotdb_parms="$iotdb_parms -Dname=iotdb\.IoTDB"
iotdb_parms="$iotdb_parms -DIOTDB_LOG_DIR=${IOTDB_LOG_DIR}"
+ iotdb_parms="$iotdb_parms -DOFF_HEAP_MEMORY=${OFF_HEAP_MEMORY}"
Review Comment:
1. lack windows
2. describe the compatibility in pr
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/memory/SystemInfo.java:
##
@@ -352,6 +378,12 @@ public long getMemorySizeForCompaction() {
}
public void allocateWriteMemory() {
+totalDirectBufferMemorySizeLimit =
+config.getMaxOffHeapMemoryBytes() == 0
+? Long.MAX_VALUE
Review Comment:
add some comments for this default value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Modify the strategy of compaction schedule [iotdb]
OneSizeFitsQuorum commented on code in PR #12469:
URL: https://github.com/apache/iotdb/pull/12469#discussion_r1590787458
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/InnerSpaceCompactionTask.java:
##
@@ -510,6 +511,13 @@ public int getProcessedFileNum() {
return selectedTsFileResourceList.size();
}
+ @Override
+ public void handleTaskCleanup() {
+if (sumOfCompactionCount == 0) {
Review Comment:
decrement when pop queue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] remove connector it [iotdb]
HTHou merged PR #12471: URL: https://github.com/apache/iotdb/pull/12471 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Fix]. add alter view statement in snapshot parser. [iotdb]
JackieTien97 merged PR #12460: URL: https://github.com/apache/iotdb/pull/12460 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] remove connector it [iotdb]
CritasWang opened a new pull request, #12471: URL: https://github.com/apache/iotdb/pull/12471 based on https://github.com/apache/iotdb/pull/12373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Pipe: Fixed connector lifeCycle to allow restarted DataRegions to reuse connectors from other not restarted DataRegions [iotdb]
Caideyipi opened a new pull request, #12470: URL: https://github.com/apache/iotdb/pull/12470 ## Description As the title said. This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Modify the strategy of compaction schedule [iotdb]
shuwenwei opened a new pull request, #12469: URL: https://github.com/apache/iotdb/pull/12469 ## Description Modify the strategy of compaction schedule. 1. Skip compaction selection if the queue is full. 2. Skip high level inner compaction when there are zero level inner compaction in compaction queue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Pipe: Return directly when SQL retry failed in IT to avoid potential fake failure interference [iotdb]
Caideyipi opened a new pull request, #12468: URL: https://github.com/apache/iotdb/pull/12468 ## Description As the title said. This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Subscription: randomly generate consumer group id and consumer id if not exist & validate topic config ahead on DN [iotdb]
VGalaxies opened a new pull request, #12467: URL: https://github.com/apache/iotdb/pull/12467 As title. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2095302936 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2095272672 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1590580947
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
Done.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/LocalExecutionPlanContext.java:
##
@@ -91,6 +98,10 @@ public class LocalExecutionPlanContext {
// use AtomicReference not for thread-safe, just for updating same field in
different pipeline
private AtomicReference> timePartitions = new AtomicReference<>();
+ /** Records the parent of each pipeline. The order of each list does not
matter for now. */
+ private Map>
parentPlanNodeIdToMemoryEstimator =
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1590580899
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
Removed.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1590580783
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
done
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2095261780 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
