Re: [PR] Take snapshot for mods using copy instead of hard link [iotdb]
codecov-commenter commented on PR #11728: URL: https://github.com/apache/iotdb/pull/11728#issuecomment-1857433569 ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/11728?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: `9 lines` in your changes are missing coverage. Please review. > Comparison is base [(`8854954`)](https://app.codecov.io/gh/apache/iotdb/commit/88549541f6bf53717169294a2fe2b222d05ddede?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 49.09% compared to head [(`c9156d8`)](https://app.codecov.io/gh/apache/iotdb/pull/11728?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 49.09%. > Report is 2 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/iotdb/pull/11728?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [...orageengine/dataregion/snapshot/SnapshotTaker.java](https://app.codecov.io/gh/apache/iotdb/pull/11728?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL3NuYXBzaG90L1NuYXBzaG90VGFrZXIuamF2YQ==) | 0.00% | [9 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/11728?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #11728 +/- ## - Coverage 49.09% 49.09% -0.01% Complexity2500825008 Files 2827 2827 Lines177064 177072 +8 Branches 2122521226 +1 - Hits 8693586934 -1 - Misses9012990138 +9 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/11728?src=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] IT: enable REST service when needed to fix IT [iotdb]
SteveYurongSu merged PR #11726: URL: https://github.com/apache/iotdb/pull/11726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Take snapshot for mods using copy instead of hard link [iotdb]
sonarcloud[bot] commented on PR #11728: URL: https://github.com/apache/iotdb/pull/11728#issuecomment-1857392475 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=11728) **Quality Gate passed** Kudos, no new issues were introduced! [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb=11728=false=true) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb=11728=false=true) [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11728=new_coverage=list) [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11728=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=11728) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [IOTDB-5706] Data inconsistency between IoT protocol replications [iotdb]
OneSizeFitsQuorum merged PR #11727: URL: https://github.com/apache/iotdb/pull/11727 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Take snapshot for mods using copy instead of hard link [iotdb]
THUMarkLau opened a new pull request, #11728: URL: https://github.com/apache/iotdb/pull/11728 Currently, a hard link of mods file is created when taking snapshot for data region, but the mods file is not immutable, which means the snapshot could be updated. We need to make sure the snapshot is immutable too, so a copy of mods file should be created instead of hard link. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: fix losing points when enable batch mode (WIP) [iotdb]
DanielWang2035 commented on PR #11722: URL: https://github.com/apache/iotdb/pull/11722#issuecomment-1857372508 LGTM. The reason is that one connector may transfer events from multiple dataregions, and events from different dataregions may have the same commit id. So we should use equals() instead of comparing commit ids. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implement Cluster ID [iotdb]
CRZbulabula commented on code in PR #11702:
URL: https://github.com/apache/iotdb/pull/11702#discussion_r1427566999
##
integration-test/src/test/java/org/apache/iotdb/confignode/it/cluster/IoTDBClusterStartIT.java:
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.it.cluster;
+
+import org.apache.iotdb.commons.client.exception.ClientManagerException;
+import org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient;
+import org.apache.iotdb.confignode.rpc.thrift.TGetClusterIdResp;
+import org.apache.iotdb.consensus.ConsensusFactory;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestLogger;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.ClusterIT;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.slf4j.Logger;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({ClusterIT.class})
+public class IoTDBClusterStartIT {
+ private static final Logger logger = IoTDBTestLogger.logger;
Review Comment:
But normally we use LoggerFactory.getLogger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] [IOTDB-5706] Data inconsistency between IoT protocol replications [iotdb]
BUAAserein opened a new pull request, #11727: URL: https://github.com/apache/iotdb/pull/11727 ## Description The status code returned for handling BatchProcessException is `WRITE_PROCESS_ERROR`, but this status code was not included in the retry judgment of IoT, so the batch that caused BatchProcessException did not retry, resulting in inconsistent data between IoT replications. To fix the problem, we add the status code `WRITE_PROCESS_ERROR` in IoT Retry judgment. Besides, the atomicity issue of storage engine write operations can also lead to data inconsistency, which will be fixed in subsequent work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implement Cluster ID [iotdb]
liyuheng5 commented on PR #11702: URL: https://github.com/apache/iotdb/pull/11702#issuecomment-1857273666 > PTAL. By the way, it's better to test version updation for this PR Checked, it's work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] SeriesScanUtil support predicate pushdown [iotdb]
sonarcloud[bot] commented on PR #11671: URL: https://github.com/apache/iotdb/pull/11671#issuecomment-1857269813 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=11671) **Quality Gate failed** Failed conditions [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11671=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=11671) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] fixed rest it [iotdb]
CloudWise-Lukemiao opened a new pull request, #11726: URL: https://github.com/apache/iotdb/pull/11726 fixed rest it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] SeriesScanUtil support predicate pushdown [iotdb]
liuminghui233 commented on code in PR #11671:
URL: https://github.com/apache/iotdb/pull/11671#discussion_r1427536493
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/SeriesScanUtil.java:
##
@@ -1192,8 +1221,8 @@ TsBlock getAllSatisfiedPageData(boolean ascending) throws
IOException {
}
}
-void setFilter(Filter filter) {
- data.setFilter(filter);
+void setPushDownFilter(Filter pushDownFilter) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] SeriesScanUtil support predicate pushdown [iotdb]
liuminghui233 commented on code in PR #11671:
URL: https://github.com/apache/iotdb/pull/11671#discussion_r1427536199
##
iotdb-core/tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/operator/Not.java:
##
@@ -57,6 +58,11 @@ public boolean satisfyRow(long time, Object[] values) {
return !filter.satisfyRow(time, values);
}
+ @Override
+ public boolean[] satisfyTsBlock(TsBlock tsBlock) {
+return filter.satisfyTsBlock(tsBlock);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Implement Cluster ID [iotdb]
liyuheng5 commented on code in PR #11702:
URL: https://github.com/apache/iotdb/pull/11702#discussion_r1427535358
##
integration-test/src/test/java/org/apache/iotdb/confignode/it/cluster/IoTDBClusterStartIT.java:
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.it.cluster;
+
+import org.apache.iotdb.commons.client.exception.ClientManagerException;
+import org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient;
+import org.apache.iotdb.confignode.rpc.thrift.TGetClusterIdResp;
+import org.apache.iotdb.consensus.ConsensusFactory;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestLogger;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.ClusterIT;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.slf4j.Logger;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({ClusterIT.class})
+public class IoTDBClusterStartIT {
+ private static final Logger logger = IoTDBTestLogger.logger;
+
+ private static final String ratisConsensusProtocolClass =
+ "org.apache.iotdb.consensus.ratis.RatisConsensus";
+
+ private static final int testConfigNodeNum = 3, testDataNodeNum = 0;
+
+ private static final long maxTestTime = TimeUnit.SECONDS.toMillis(30);
+ private static final long testInterval = TimeUnit.SECONDS.toMillis(1);
+
+ @Before
+ public void setUp() {
+EnvFactory.getEnv()
+.getConfig()
+.getCommonConfig()
+.setConfigNodeConsensusProtocolClass(ratisConsensusProtocolClass)
+.setSchemaRegionConsensusProtocolClass(ratisConsensusProtocolClass)
+.setDataRegionConsensusProtocolClass(ratisConsensusProtocolClass)
+.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
+
+EnvFactory.getEnv().initClusterEnvironment(testConfigNodeNum,
testDataNodeNum);
+ }
+
+ @After
+ public void tearDown() {
+EnvFactory.getEnv().cleanClusterEnvironment();
+ }
+
+ @Test
+ public void clusterIdTest() {
+try (SyncConfigNodeIServiceClient client =
+(SyncConfigNodeIServiceClient)
EnvFactory.getEnv().getLeaderConfigNodeConnection()) {
+ long startTime = System.currentTimeMillis();
+ while (System.currentTimeMillis() - startTime < maxTestTime) {
+try {
+ TGetClusterIdResp resp = client.getClusterId();
+ if (TSStatusCode.SUCCESS_STATUS.getStatusCode() ==
resp.getStatus().getCode()) {
+return;
+ }
+} catch (TException ignore) {
+
+}
Review Comment:
indeed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Implement Cluster ID [iotdb]
liyuheng5 commented on code in PR #11702:
URL: https://github.com/apache/iotdb/pull/11702#discussion_r1427534940
##
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/request/write/confignode/UpdateClusterIdPlan.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.consensus.request.write.confignode;
+
+import org.apache.iotdb.confignode.consensus.request.ConfigPhysicalPlan;
+import org.apache.iotdb.confignode.consensus.request.ConfigPhysicalPlanType;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Objects;
+
+public class UpdateClusterIdPlan extends ConfigPhysicalPlan {
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] SeriesScanUtil support predicate pushdown [iotdb]
liuminghui233 commented on code in PR #11671:
URL: https://github.com/apache/iotdb/pull/11671#discussion_r1427534869
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/SeriesScanUtil.java:
##
@@ -1097,12 +1139,7 @@ private void unpackSeqTsFileResource() throws
IOException {
private void unpackUnseqTsFileResource() throws IOException {
ITimeSeriesMetadata timeseriesMetadata =
-loadTimeSeriesMetadata(
-orderUtils.getNextUnseqFileResource(true),
-seriesPath,
-context,
-getGlobalTimeFilter(),
-scanOptions.getAllSensors());
+loadTimeSeriesMetadata(orderUtils.getNextUnseqFileResource(true),
seriesPath, context);
if (timeseriesMetadata != null) {
timeseriesMetadata.setModified(true);
Review Comment:
fixed
##
iotdb-core/tsfile/src/main/java/org/apache/iotdb/tsfile/read/common/block/TsBlock.java:
##
@@ -223,6 +223,18 @@ public Column[] getColumns(int[] columnIndexes) {
return columns;
}
+ public Object[] getRowValues(int rowIndex) {
+Object[] rowValues = new Object[valueColumns.length];
+for (int i = 0; i < valueColumns.length; i++) {
+ if (valueColumns[i].isNull(rowIndex)) {
+rowValues[i] = null;
+ } else {
+rowValues[i] = valueColumns[i].getObject(rowIndex);
+ }
+}
+return rowValues;
+ }
+
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] SeriesScanUtil support predicate pushdown [iotdb]
liuminghui233 commented on code in PR #11671:
URL: https://github.com/apache/iotdb/pull/11671#discussion_r1427534631
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/SeriesScanUtil.java:
##
@@ -,26 +1148,29 @@ private void unpackUnseqTsFileResource() throws
IOException {
}
protected ITimeSeriesMetadata loadTimeSeriesMetadata(
- TsFileResource resource,
- PartialPath seriesPath,
- QueryContext context,
- Filter filter,
- Set allSensors)
- throws IOException {
+ TsFileResource resource, PartialPath seriesPath, QueryContext context)
throws IOException {
return FileLoaderUtils.loadTimeSeriesMetadata(
-resource, seriesPath, context, filter, allSensors);
+resource,
+seriesPath,
+context,
+scanOptions.getGlobalTimeFilter(),
+scanOptions.getAllSensors());
}
- protected List getTsDataTypeList() {
+ public List getTsDataTypeList() {
return Collections.singletonList(dataType);
}
protected IPointReader getPointReader(TsBlock tsBlock) {
return tsBlock.getTsBlockSingleColumnIterator();
}
- Filter getGlobalTimeFilter() {
-return scanOptions.getGlobalTimeFilter();
+ protected boolean timeAllSelected(IMetadata metadata) {
+return metadata.timeAllSelected();
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Implement Cluster ID [iotdb]
liyuheng5 commented on code in PR #11702:
URL: https://github.com/apache/iotdb/pull/11702#discussion_r1427532888
##
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/ClusterInfo.java:
##
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.persistence;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.commons.snapshot.SnapshotProcessor;
+import
org.apache.iotdb.confignode.consensus.request.write.confignode.UpdateClusterIdPlan;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.UUID;
+
+public class ClusterInfo implements SnapshotProcessor {
Review Comment:
good point !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] SeriesScanUtil support predicate pushdown [iotdb]
JackieTien97 commented on code in PR #11671:
URL: https://github.com/apache/iotdb/pull/11671#discussion_r1420048752
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/SeriesScanUtil.java:
##
@@ -,26 +1148,29 @@ private void unpackUnseqTsFileResource() throws
IOException {
}
protected ITimeSeriesMetadata loadTimeSeriesMetadata(
- TsFileResource resource,
- PartialPath seriesPath,
- QueryContext context,
- Filter filter,
- Set allSensors)
- throws IOException {
+ TsFileResource resource, PartialPath seriesPath, QueryContext context)
throws IOException {
return FileLoaderUtils.loadTimeSeriesMetadata(
-resource, seriesPath, context, filter, allSensors);
+resource,
+seriesPath,
+context,
+scanOptions.getGlobalTimeFilter(),
+scanOptions.getAllSensors());
}
- protected List getTsDataTypeList() {
+ public List getTsDataTypeList() {
return Collections.singletonList(dataType);
}
protected IPointReader getPointReader(TsBlock tsBlock) {
return tsBlock.getTsBlockSingleColumnIterator();
}
- Filter getGlobalTimeFilter() {
-return scanOptions.getGlobalTimeFilter();
+ protected boolean timeAllSelected(IMetadata metadata) {
+return metadata.timeAllSelected();
Review Comment:
just return true? since AlignedSeriesScanUtil already override this method.
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/SeriesScanUtil.java:
##
@@ -1097,12 +1139,7 @@ private void unpackSeqTsFileResource() throws
IOException {
private void unpackUnseqTsFileResource() throws IOException {
ITimeSeriesMetadata timeseriesMetadata =
-loadTimeSeriesMetadata(
-orderUtils.getNextUnseqFileResource(true),
-seriesPath,
-context,
-getGlobalTimeFilter(),
-scanOptions.getAllSensors());
+loadTimeSeriesMetadata(orderUtils.getNextUnseqFileResource(true),
seriesPath, context);
if (timeseriesMetadata != null) {
timeseriesMetadata.setModified(true);
Review Comment:
why here need to be set to modified?
##
iotdb-core/tsfile/src/main/java/org/apache/iotdb/tsfile/read/common/block/TsBlock.java:
##
@@ -223,6 +223,18 @@ public Column[] getColumns(int[] columnIndexes) {
return columns;
}
+ public Object[] getRowValues(int rowIndex) {
+Object[] rowValues = new Object[valueColumns.length];
+for (int i = 0; i < valueColumns.length; i++) {
+ if (valueColumns[i].isNull(rowIndex)) {
+rowValues[i] = null;
+ } else {
+rowValues[i] = valueColumns[i].getObject(rowIndex);
+ }
+}
+return rowValues;
+ }
+
Review Comment:
```suggestion
```
not used, can be removed.
##
iotdb-core/tsfile/src/main/java/org/apache/iotdb/tsfile/read/filter/operator/Not.java:
##
@@ -57,6 +58,11 @@ public boolean satisfyRow(long time, Object[] values) {
return !filter.satisfyRow(time, values);
}
+ @Override
+ public boolean[] satisfyTsBlock(TsBlock tsBlock) {
+return filter.satisfyTsBlock(tsBlock);
Review Comment:
should add `!`
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/SeriesScanUtil.java:
##
@@ -1192,8 +1221,8 @@ TsBlock getAllSatisfiedPageData(boolean ascending) throws
IOException {
}
}
-void setFilter(Filter filter) {
- data.setFilter(filter);
+void setPushDownFilter(Filter pushDownFilter) {
Review Comment:
```suggestion
void addPushDownFilter(Filter pushDownFilter) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Implement Cluster ID [iotdb]
liyuheng5 commented on code in PR #11702:
URL: https://github.com/apache/iotdb/pull/11702#discussion_r1427519409
##
integration-test/src/test/java/org/apache/iotdb/confignode/it/cluster/IoTDBClusterStartIT.java:
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.it.cluster;
+
+import org.apache.iotdb.commons.client.exception.ClientManagerException;
+import org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient;
+import org.apache.iotdb.confignode.rpc.thrift.TGetClusterIdResp;
+import org.apache.iotdb.consensus.ConsensusFactory;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestLogger;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.ClusterIT;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.slf4j.Logger;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({ClusterIT.class})
+public class IoTDBClusterStartIT {
+ private static final Logger logger = IoTDBTestLogger.logger;
Review Comment:
To follow IoTDBClusterRestartIT :P
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Implement Cluster ID [iotdb]
liyuheng5 commented on code in PR #11702:
URL: https://github.com/apache/iotdb/pull/11702#discussion_r1427518489
##
integration-test/src/test/java/org/apache/iotdb/confignode/it/cluster/IoTDBClusterStartIT.java:
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.it.cluster;
+
+import org.apache.iotdb.commons.client.exception.ClientManagerException;
+import org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient;
+import org.apache.iotdb.confignode.rpc.thrift.TGetClusterIdResp;
+import org.apache.iotdb.consensus.ConsensusFactory;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestLogger;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.ClusterIT;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.slf4j.Logger;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({ClusterIT.class})
+public class IoTDBClusterStartIT {
+ private static final Logger logger = IoTDBTestLogger.logger;
+
+ private static final String ratisConsensusProtocolClass =
+ "org.apache.iotdb.consensus.ratis.RatisConsensus";
+
+ private static final int testConfigNodeNum = 3, testDataNodeNum = 0;
+
+ private static final long maxTestTime = TimeUnit.SECONDS.toMillis(30);
+ private static final long testInterval = TimeUnit.SECONDS.toMillis(1);
+
+ @Before
+ public void setUp() {
+EnvFactory.getEnv()
+.getConfig()
+.getCommonConfig()
+.setConfigNodeConsensusProtocolClass(ratisConsensusProtocolClass)
+.setSchemaRegionConsensusProtocolClass(ratisConsensusProtocolClass)
+.setDataRegionConsensusProtocolClass(ratisConsensusProtocolClass)
+.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
+
+EnvFactory.getEnv().initClusterEnvironment(testConfigNodeNum,
testDataNodeNum);
+ }
+
+ @After
+ public void tearDown() {
+EnvFactory.getEnv().cleanClusterEnvironment();
+ }
+
+ @Test
+ public void clusterIdTest() {
+try (SyncConfigNodeIServiceClient client =
+(SyncConfigNodeIServiceClient)
EnvFactory.getEnv().getLeaderConfigNodeConnection()) {
+ long startTime = System.currentTimeMillis();
+ while (System.currentTimeMillis() - startTime < maxTestTime) {
+try {
+ TGetClusterIdResp resp = client.getClusterId();
+ if (TSStatusCode.SUCCESS_STATUS.getStatusCode() ==
resp.getStatus().getCode()) {
+return;
+ }
+} catch (TException ignore) {
+
+}
+ }
+ String errorMessage = String.format("Cluster ID failed to generate in %d
ms.", maxTestTime);
+ Assert.fail(errorMessage);
+} catch (ClientManagerException | IOException | InterruptedException e) {
+ throw new RuntimeException(e);
Review Comment:
Sure, now throw them to upper level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Add a param to switch whether to enable data separation or not [iotdb]
xingtanzjr merged PR #11692: URL: https://github.com/apache/iotdb/pull/11692 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implement Cluster ID [iotdb]
liyuheng5 commented on code in PR #11702:
URL: https://github.com/apache/iotdb/pull/11702#discussion_r1427517532
##
integration-test/src/test/java/org/apache/iotdb/confignode/it/cluster/IoTDBClusterStartIT.java:
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.it.cluster;
+
+import org.apache.iotdb.commons.client.exception.ClientManagerException;
+import org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient;
+import org.apache.iotdb.confignode.rpc.thrift.TGetClusterIdResp;
+import org.apache.iotdb.consensus.ConsensusFactory;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestLogger;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.ClusterIT;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.slf4j.Logger;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({ClusterIT.class})
+public class IoTDBClusterStartIT {
+ private static final Logger logger = IoTDBTestLogger.logger;
+
+ private static final String ratisConsensusProtocolClass =
+ "org.apache.iotdb.consensus.ratis.RatisConsensus";
+
+ private static final int testConfigNodeNum = 3, testDataNodeNum = 0;
+
+ private static final long maxTestTime = TimeUnit.SECONDS.toMillis(30);
+ private static final long testInterval = TimeUnit.SECONDS.toMillis(1);
+
+ @Before
+ public void setUp() {
+EnvFactory.getEnv()
+.getConfig()
+.getCommonConfig()
+.setConfigNodeConsensusProtocolClass(ratisConsensusProtocolClass)
+.setSchemaRegionConsensusProtocolClass(ratisConsensusProtocolClass)
+.setDataRegionConsensusProtocolClass(ratisConsensusProtocolClass)
+.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
+
+EnvFactory.getEnv().initClusterEnvironment(testConfigNodeNum,
testDataNodeNum);
+ }
+
+ @After
+ public void tearDown() {
+EnvFactory.getEnv().cleanClusterEnvironment();
+ }
+
+ @Test
+ public void clusterIdTest() {
+try (SyncConfigNodeIServiceClient client =
+(SyncConfigNodeIServiceClient)
EnvFactory.getEnv().getLeaderConfigNodeConnection()) {
+ long startTime = System.currentTimeMillis();
+ while (System.currentTimeMillis() - startTime < maxTestTime) {
+try {
+ TGetClusterIdResp resp = client.getClusterId();
+ if (TSStatusCode.SUCCESS_STATUS.getStatusCode() ==
resp.getStatus().getCode()) {
+return;
+ }
Review Comment:
omg sorry I forgot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Optimize InsertRecords performance by reducing record metrics [iotdb]
codecov-commenter commented on PR #11723: URL: https://github.com/apache/iotdb/pull/11723#issuecomment-1857179774 ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/11723?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: `72 lines` in your changes are missing coverage. Please review. > Comparison is base [(`60d66ae`)](https://app.codecov.io/gh/apache/iotdb/commit/60d66ae873fc7f9c60206a84ce6985ecec672d53?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 49.08% compared to head [(`cc0973d`)](https://app.codecov.io/gh/apache/iotdb/pull/11723?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 49.09%. > Report is 23 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/iotdb/pull/11723?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [.../iotdb/db/storageengine/dataregion/DataRegion.java](https://app.codecov.io/gh/apache/iotdb/pull/11723?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL0RhdGFSZWdpb24uamF2YQ==) | 16.47% | [71 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/11723?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [.../statemachine/dataregion/DataExecutionVisitor.java](https://app.codecov.io/gh/apache/iotdb/pull/11723?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvY29uc2Vuc3VzL3N0YXRlbWFjaGluZS9kYXRhcmVnaW9uL0RhdGFFeGVjdXRpb25WaXNpdG9yLmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/11723?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@Coverage Diff @@ ## master #11723+/- ## == Coverage 49.08% 49.09% - Complexity2497025025+55 == Files 2816 2827+11 Lines176696 177168 +472 Branches 2118121239+58 == + Hits 8673086973 +243 - Misses8996690195 +229 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/11723?src=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Revert "add IT testing to rest service (#11437)" [iotdb]
sonarcloud[bot] commented on PR #11725: URL: https://github.com/apache/iotdb/pull/11725#issuecomment-1857173403 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=11725) **Quality Gate passed** Kudos, no new issues were introduced! [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb=11725=false=true) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb=11725=false=true) No data about Coverage [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11725=new_duplicated_lines_density=list) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=11725) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Revert "Pipe Schema: Receiver Agent: Added pipe enriched planNode to enable pipe request detection to configure "forwarding-pipe-request" (#11672)" [iotdb]
sonarcloud[bot] commented on PR #11724: URL: https://github.com/apache/iotdb/pull/11724#issuecomment-1857167709 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=11724) **Quality Gate failed** Failed conditions [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11724=new_coverage=list) (required ≥ 80%) [3.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11724=new_duplicated_lines_density=list) (required ≤ 3%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=11724) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Revert "add IT testing to rest service (#11437)" [iotdb]
SteveYurongSu opened a new pull request, #11725: URL: https://github.com/apache/iotdb/pull/11725 This reverts commit f12b5eb5f517184819086a2acd57af076b272875. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Improved stability of Ratis addMemberToGroup and testThresholdSnapshot UT [iotdb]
OneSizeFitsQuorum merged PR #11717: URL: https://github.com/apache/iotdb/pull/11717 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Revert "Pipe Schema: Receiver Agent: Added pipe enriched planNode to enable pipe request detection to configure "forwarding-pipe-request" (#11672)" [iotdb]
SteveYurongSu closed pull request #11724: Revert "Pipe Schema: Receiver Agent: Added pipe enriched planNode to enable pipe request detection to configure "forwarding-pipe-request" (#11672)" URL: https://github.com/apache/iotdb/pull/11724 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Revert "Pipe Schema: Receiver Agent: Added pipe enriched planNode to enable pipe request detection to configure "forwarding-pipe-request" (#11672)" [iotdb]
SteveYurongSu opened a new pull request, #11724: URL: https://github.com/apache/iotdb/pull/11724 This reverts commit 3e4c6a4df918513cb1c002614da3837090a4dfc8. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Optimize InsertRecords performance by reducing record metrics [iotdb]
sonarcloud[bot] commented on PR #11723: URL: https://github.com/apache/iotdb/pull/11723#issuecomment-1857155524 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=11723) **Quality Gate failed** Failed conditions [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11723=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=11723) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Optimize InsertRecords performance by reducing record metrics [iotdb]
HTHou opened a new pull request, #11723: URL: https://github.com/apache/iotdb/pull/11723 ## Description https://apache-iotdb.feishu.cn/docx/AlXqdy9PjoWoenx1WfBcjNOqnyg#JAN1dkRu9o2B4xxVVwfc4V7onze  In the insertRecords interface, the record scheduleLockCost occupies 2.2% of the global CPU overhead, while it is only 0.03% in insertTablet. One important reason for this is that when insertRows is executed, it iterates over Rows, applying insertRow once to each Row. Each time insertRow is called, Metrics will update cost. When there are very many rows, the overhead of updating the Histogram becomes significant. The simplest optimization is to return the cost as a result to the upper layer, and then update the Cost uniformly once all rows have been inserted. ## Test Benchmark config ``` DB_SWITCH=IoTDB-110-SESSION_BY_RECORDS LOOP=10 DEVICE_NUMBER=5000 SENSOR_NUMBER=20 VECTOR=false BATCH_SIZE_PER_WRITE=1000 ``` IOTDB config ``` Confignode ON_HEAP_MEMORY=8G Datanode ON_HEAP_MEMORY=8G ``` Before the optimization Throughput 8367755.72 point/s Profile https://github.com/apache/iotdb/assets/25913899/06c26b50-343f-4dc4-ba48-24b70c511524;> After the optimization Throughput 6751503.68 point/s Profile https://github.com/apache/iotdb/assets/25913899/f2a3c708-007b-4ad2-bd11-2b138d1949f6;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Running compaction task failed [iotdb]
zhaozhihua2008 closed issue #11600: [Bug] Running compaction task failed URL: https://github.com/apache/iotdb/issues/11600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Running compaction task failed [iotdb]
zhaozhihua2008 commented on issue #11600: URL: https://github.com/apache/iotdb/issues/11600#issuecomment-1857128467 谢谢。手动执行运维命令，flush，merge。之后无此问题了。 主要我们的路径中有数字，升级到1.0 报错，暂时无法升级。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] PBTree Improvement: concurrent operation and interleaved flush [iotdb]
MarcosZyk merged PR #11698: URL: https://github.com/apache/iotdb/pull/11698 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Pipe: fix losing points when enable batch mode (WIP) [iotdb]
VGalaxies opened a new pull request, #11722: URL: https://github.com/apache/iotdb/pull/11722 ## Description As title, the specific reason are not yet clear... This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [x] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe Schema: Receiver Agent: Added pipe enriched planNode to enable pipe request detection to configure "forwarding-pipe-request" [iotdb]
SteveYurongSu merged PR #11672: URL: https://github.com/apache/iotdb/pull/11672 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: enhance the idempotence of start / stop pipe [iotdb]
Caideyipi commented on code in PR #11713:
URL: https://github.com/apache/iotdb/pull/11713#discussion_r1426496383
##
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/pipe/task/PipeTaskCoordinator.java:
##
@@ -120,15 +120,9 @@ public TSStatus startPipe(String pipeName) {
public TSStatus stopPipe(String pipeName) {
final boolean isStoppedByRuntimeException =
pipeTaskInfo.isStoppedByRuntimeException(pipeName);
final TSStatus status =
configManager.getProcedureManager().stopPipe(pipeName);
-if (status == RpcUtils.SUCCESS_STATUS) {
+if (status.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
Review Comment:
This can be leaved aside since I have another PR which cuts this out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] fix bad url of V1.3 and master [iotdb-docs]
qiaojialin merged PR #133: URL: https://github.com/apache/iotdb-docs/pull/133 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix bad url of V1.3 and master [iotdb-docs]
StefanieZhao7 commented on code in PR #133:
URL: https://github.com/apache/iotdb-docs/pull/133#discussion_r1426549680
##
src/.vuepress/sidebar_timecho/V1.3.x/zh.ts:
##
@@ -89,6 +89,7 @@ export const zhSidebar = {
{ text: '数据同步', link: 'Data-Sync_timecho' },
{ text: '多级存储', link: 'Tiered-Storage_timecho' },
{ text: '视图', link: 'IoTDB-View_timecho' },
+{ text: 'IoTDB原生机器学习', link: 'IoTDB-AINode_timecho' },
Review Comment:
叫"AINode机器学习框架"吧
##
src/.vuepress/sidebar_timecho/V1.3.x/zh.ts:
##
@@ -89,6 +89,7 @@ export const zhSidebar = {
{ text: '数据同步', link: 'Data-Sync_timecho' },
{ text: '多级存储', link: 'Tiered-Storage_timecho' },
{ text: '视图', link: 'IoTDB-View_timecho' },
+{ text: 'IoTDB原生机器学习', link: 'IoTDB-AINode_timecho' },
Review Comment:
叫"AINode机器学习框架"吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Pipe: validate the value of extractor.pattern [iotdb]
Caideyipi commented on code in PR #11720:
URL: https://github.com/apache/iotdb/pull/11720#discussion_r1426544815
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/IoTDBDataRegionExtractor.java:
##
@@ -84,6 +88,14 @@ public IoTDBDataRegionExtractor() {
@Override
public void validate(PipeParameterValidator validator) throws Exception {
+// Check whether the pattern is legal
+PathUtils.isLegalPath(
Review Comment:
A legal prefix may not be a legal path.. Such as root.adc.1, can match
root.adc.1ts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Pipe: validate the value of extractor.pattern [iotdb]
Caideyipi commented on code in PR #11720:
URL: https://github.com/apache/iotdb/pull/11720#discussion_r1426532193
##
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/pipe/extractor/IoTDBDataRegionExtractorTest.java:
##
@@ -53,4 +54,21 @@ public void testIoTDBDataRegionExtractor() {
Assert.fail();
}
}
+
+ @Test
+ public void testIoTDBDataRegionExtractorWithIllegalPattern() {
+IoTDBDataRegionExtractor extractor = new IoTDBDataRegionExtractor();
+try {
+ extractor.validate(
+ new PipeParameterValidator(
+ new PipeParameters(
+ new HashMap() {
+{
+ put(PipeExtractorConstant.EXTRACTOR_PATTERN_KEY,
"root.a-b");
+}
+ })));
+} catch (Exception e) {
+ Assert.assertEquals(e.getClass(), IllegalPathException.class);
Review Comment:
@Test(expected = IllegalArgumentException.class)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Pipe: rename iotdb-thrift-ssl-sink & allow not specifying connector in kv attrs & support kv attrs that don't have sink. / processor. / sink. prefixes [iotdb]
SteveYurongSu merged PR #11715: URL: https://github.com/apache/iotdb/pull/11715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Show devices with template [iotdb]
linxt20 commented on code in PR #11681:
URL: https://github.com/apache/iotdb/pull/11681#discussion_r1426469401
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/utils/filter/DeviceFilterVisitor.java:
##
@@ -38,4 +40,26 @@ public boolean visitPathContainsFilter(
}
return
info.getFullPath().toLowerCase().contains(pathContainsFilter.getContainString());
}
+
+ @Override
+ public boolean visitTemplateFilter(TemplateFilter templateFilter,
IDeviceSchemaInfo info) {
+if (templateFilter.getTemplateName() == null) {
+ return true;
+}
+String TemplateName = "null";
+boolean EqualAns;
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Show devices with template [iotdb]
linxt20 commented on code in PR #11681:
URL: https://github.com/apache/iotdb/pull/11681#discussion_r1426449145
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/utils/filter/DeviceFilterVisitor.java:
##
@@ -38,4 +40,26 @@ public boolean visitPathContainsFilter(
}
return
info.getFullPath().toLowerCase().contains(pathContainsFilter.getContainString());
}
+
+ @Override
+ public boolean visitTemplateFilter(TemplateFilter templateFilter,
IDeviceSchemaInfo info) {
+if (templateFilter.getTemplateName() == null) {
+ return true;
+}
+String TemplateName = "null";
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Pipe: iotdb-thrift-ssl-sink & allow not specifying connector in kv attrs & support kv attrs that dont have sink. processor. sink. prefixes [iotdb]
sonarcloud[bot] commented on PR #11715: URL: https://github.com/apache/iotdb/pull/11715#issuecomment-1855433776 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=11715) **Quality Gate failed** Failed conditions [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11715=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=11715) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: validate the value of extractor.pattern [iotdb]
SteveYurongSu commented on code in PR #11720:
URL: https://github.com/apache/iotdb/pull/11720#discussion_r1426390154
##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/IoTDBDataRegionExtractor.java:
##
@@ -84,6 +87,11 @@ public IoTDBDataRegionExtractor() {
@Override
public void validate(PipeParameterValidator validator) throws Exception {
+// Check whether the pattern is legal
+PathUtils.isLegalPath(
+validator
+.getParameters()
+.getStringOrDefault(EXTRACTOR_PATTERN_KEY,
EXTRACTOR_PATTERN_DEFAULT_VALUE));
Review Comment:
```suggestion
// Check whether the pattern is legal
PathUtils.isLegalPath(
validator
.getParameters()
.getStringOrDefault(EXTRACTOR_PATTERN_KEY,
EXTRACTOR_PATTERN_DEFAULT_VALUE));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Pipe: iotdb-thrift-ssl-sink & allow not specifying connector in kv attrs & support kv attrs that dont have sink. processor. sink. prefixes [iotdb]
sonarcloud[bot] commented on PR #11715: URL: https://github.com/apache/iotdb/pull/11715#issuecomment-1855383486 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=11715) **Quality Gate failed** Failed conditions [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11715=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=11715) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Pipe: validate the value of extractor.pattern [iotdb]
xuanronaldo opened a new pull request, #11720: URL: https://github.com/apache/iotdb/pull/11720 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: iotdb-thrift-ssl-sink & allow not specifying connector in kv attrs & support kv attrs that dont have sink. processor. sink. prefixes [iotdb]
sonarcloud[bot] commented on PR #11715: URL: https://github.com/apache/iotdb/pull/11715#issuecomment-1855362517 ## [](https://sonarcloud.io/dashboard?id=apache_iotdb=11715) **Quality Gate failed** Failed conditions [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=11715=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=11715) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
