Re: [PR] Enhance estimate of FI memory usage [iotdb]
JackieTien97 merged PR #12393: URL: https://github.com/apache/iotdb/pull/12393 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2098197224 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1592317226
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/MemoryMeasurable.java:
##
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution;
+
+public interface MemoryMeasurable {
Review Comment:
Changed to Accountable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1592316379
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/LastQueryOperator.java:
##
@@ -173,4 +178,15 @@ public long calculateRetainedSizeAfterCallingNext() {
}
return max;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
Review Comment:
Done. I added the size of tsBlockBuilder in all operators that have this
field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1592315583
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/UpdateLastCacheOperator.java:
##
@@ -89,4 +94,14 @@ protected void appendLastValueToTsBlockBuilder(long
lastTime, TsPrimitiveType la
LastQueryUtil.appendLastValue(
tsBlockBuilder, lastTime, fullPath.getFullPath(),
lastValue.getStringValue(), dataType);
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
+return INSTANCE_SIZE
Review Comment:
Now I refactored EstimatedSizeOfPartialPathWithoutClassSize to
EstimatedSizeOfPartialPath in which the INSTANCE_SIZE of PartialPath is
calculated. I removed INSTANCE_SIZE of PartialPath in other classes.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/LastQueryTransformOperator.java:
##
@@ -117,4 +122,13 @@ public void close() throws Exception {
}
tsBlockBuilder = null;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1592314017
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/OperatorTreeGenerator.java:
##
@@ -2947,6 +2949,18 @@ public List
dealWithConsumeAllChildrenPipelineBreaker(
for (PlanNode localChild : node.getChildren()) {
Operator childOperation = localChild.accept(this, context);
parentPipelineChildren.add(childOperation);
+// if we don't create extra pipeline, the root of the child pipeline
should be current root
+// for example, we have IdentitySinkNode -> DeviceViewNode ->
[ScanNode, ScanNode, ScanNode]
+// the parent of the pipeline of ScanNode should be IdentitySinkNode
in the map, otherwise
+// we will lose the information of these pipelines
+List childPipelineMemoryEstimators =
+
context.getParentPlanNodeIdToMemoryEstimator().get(localChild.getPlanNodeId());
+if (childPipelineMemoryEstimators != null) {
+
context.getParentPlanNodeIdToMemoryEstimator().remove(localChild.getPlanNodeId());
+ context
+ .getParentPlanNodeIdToMemoryEstimator()
+ .put(node.getPlanNodeId(), childPipelineMemoryEstimators);
+}
Review Comment:
I updated the comment.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/LastCacheScanOperator.java:
##
@@ -83,4 +87,11 @@ public long calculateRetainedSizeAfterCallingNext() {
public PlanNodeId getSourceId() {
return sourceId;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
+return INSTANCE_SIZE
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
JackieTien97 commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1590950784
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/LastQueryOperator.java:
##
@@ -173,4 +178,15 @@ public long calculateRetainedSizeAfterCallingNext() {
}
return max;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
Review Comment:
You should think about the memory size of `tsBlockBuilder` field.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/UpdateLastCacheOperator.java:
##
@@ -89,4 +94,14 @@ protected void appendLastValueToTsBlockBuilder(long
lastTime, TsPrimitiveType la
LastQueryUtil.appendLastValue(
tsBlockBuilder, lastTime, fullPath.getFullPath(),
lastValue.getStringValue(), dataType);
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
+return INSTANCE_SIZE
Review Comment:
where did you add the class size of `MeasurementPath` since you only add
`EstimatedSizeOfPartialPathWithoutClassSize` here.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/OperatorTreeGenerator.java:
##
@@ -2947,6 +2949,18 @@ public List
dealWithConsumeAllChildrenPipelineBreaker(
for (PlanNode localChild : node.getChildren()) {
Operator childOperation = localChild.accept(this, context);
parentPipelineChildren.add(childOperation);
+// if we don't create extra pipeline, the root of the child pipeline
should be current root
+// for example, we have IdentitySinkNode -> DeviceViewNode ->
[ScanNode, ScanNode, ScanNode]
+// the parent of the pipeline of ScanNode should be IdentitySinkNode
in the map, otherwise
+// we will lose the information of these pipelines
+List childPipelineMemoryEstimators =
+
context.getParentPlanNodeIdToMemoryEstimator().get(localChild.getPlanNodeId());
+if (childPipelineMemoryEstimators != null) {
+
context.getParentPlanNodeIdToMemoryEstimator().remove(localChild.getPlanNodeId());
+ context
+ .getParentPlanNodeIdToMemoryEstimator()
+ .put(node.getPlanNodeId(), childPipelineMemoryEstimators);
+}
Review Comment:
What does this mean, and the case is incorrect, because `DeviceViewNode`
won't appear in `dealWithConsumeAllChildrenPipelineBreaker`.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/LastCacheScanOperator.java:
##
@@ -83,4 +87,11 @@ public long calculateRetainedSizeAfterCallingNext() {
public PlanNodeId getSourceId() {
return sourceId;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
+return INSTANCE_SIZE
Review Comment:
also think about the memory usage of `tsBlock` field.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/last/LastQueryTransformOperator.java:
##
@@ -117,4 +122,13 @@ public void close() throws Exception {
}
tsBlockBuilder = null;
}
+
+ @Override
+ public long getEstimatedMemoryUsageInBytes() {
Review Comment:
also think about `tsBlockBuilder` field's memory size.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/MemoryMeasurable.java:
##
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution;
+
+public interface MemoryMeasurable {
Review Comment:
what's the difference with `Accountable` interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2095302936 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2095272672 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1590580947
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
Done.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/LocalExecutionPlanContext.java:
##
@@ -91,6 +98,10 @@ public class LocalExecutionPlanContext {
// use AtomicReference not for thread-safe, just for updating same field in
different pipeline
private AtomicReference> timePartitions = new AtomicReference<>();
+ /** Records the parent of each pipeline. The order of each list does not
matter for now. */
+ private Map>
parentPlanNodeIdToMemoryEstimator =
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1590580899
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
Removed.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
lancelly commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1590580783
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
done
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2095261780 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2095186924 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2095107647 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions  [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
JackieTien97 commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1577089745
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
EXPLAIN_ANALYZE((short) 90),
PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+ FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),
Review Comment:
what's this for?
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/LocalExecutionPlanContext.java:
##
@@ -91,6 +98,10 @@ public class LocalExecutionPlanContext {
// use AtomicReference not for thread-safe, just for updating same field in
different pipeline
private AtomicReference> timePartitions = new AtomicReference<>();
+ /** Records the parent of each pipeline. The order of each list does not
matter for now. */
+ private Map>
parentPlanNodeIdToMemoryEstimator =
Review Comment:
remember to set this map to null, after you check the memory, this Map may
occupy huge memory when there are many devices;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
