Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577310909 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java: ## @@ -0,0 +1,328 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl; + +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.exception.IllegalPathException; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.db.conf.IoTDBConfig; +import org.apache.iotdb.db.conf.IoTDBDescriptor; +import org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils; +import org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector; +import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion; +import org.apache.iotdb.db.storageengine.dataregion.modification.Modification; +import org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex; +import org.apache.iotdb.tsfile.file.metadata.IDeviceID; +import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import static org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED; + +public class SettleSelectorImpl implements ISettleSelector { + private static final Logger LOGGER = + LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME); + private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig(); + + private final boolean heavySelect; + private final String storageGroupName; + private final String dataRegionId; + private final long timePartition; + private final TsFileManager tsFileManager; + private boolean isSeq; + + public SettleSelectorImpl( + boolean heavySelect, + String storageGroupName, + String dataRegionId, + long timePartition, + TsFileManager tsFileManager) { +this.heavySelect = heavySelect; +this.storageGroupName = storageGroupName; +this.dataRegionId = dataRegionId; +this.timePartition = timePartition; +this.tsFileManager = tsFileManager; + } + + static class AllDirtyResource { +List resources = new ArrayList<>(); + +public void add(TsFileResource resource) { + resources.add(resource); +} + +public List getResources() { + return resources; +} + } + + static class PartialDirtyResource { +List resources = new ArrayList<>(); +long totalFileSize = 0; + +public boolean add(TsFileResource resource, long dirtyDataSize) { + resources.add(resource); + totalFileSize += resource.getTsFileSize(); + totalFileSize -= dirtyDataSize; + return checkHasReachedThreshold(); +} + +public List getResources() { + return resources; +} + +public boolean checkHasReachedThreshold() { + return resources.size() >= config.getFileLimitPerInnerTask() + || totalFileSize >= config.getTargetCompactionFileSize(); +} + } + + @Override + public List selectSettleTask(List tsFileResources) { +if
Re: [PR] Try to fix timezone [iotdb]
sonarcloud[bot] commented on PR #12401: URL: https://github.com/apache/iotdb/pull/12401#issuecomment-2074096303 ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12401) **Quality Gate passed** Issues ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb=12401=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_iotdb=12401=new_accepted_issues=list) Measures ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb=12401=false=true) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12401) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Added batch to schema snapshot execution in template activation and timeseries creation & Fixed the bug that the "CreateMultiTimeSeries" group with alias is not idempotent [iotdb]
SteveYurongSu merged PR #12380: URL: https://github.com/apache/iotdb/pull/12380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577222419 ## iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/CompactionSchedulerWithFastPerformerTest.java: ## @@ -154,7 +154,7 @@ public void test1() throws IOException, MetadataException, InterruptedException .setTargetCompactionFileSize(2L * 1024L * 1024L * 1024L); String sgName = COMPACTION_TEST_SG + "test1"; try { - CompactionTaskManager.getInstance().restart(); + // CompactionTaskManager.getInstance().restart(); Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Two stage aggregate counter [iotdb]
sonarcloud[bot] commented on PR #12328: URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2073942021 ## [![Quality Gate Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png 'Quality Gate Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) **Quality Gate failed** Failed conditions ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12328=new_coverage=list) (required ≥ 80%) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) (required ≥ A) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) ## ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png '') Catch issues before they fail your Quality Gate with our IDE extension ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png '') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577180788 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL; +ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL); +ttlCount = 1; + } + + /** + * Put ttl into cache tree. + * + * @param nodes should be prefix path or specific device path without wildcard + */ + public void setTTL(String[] nodes, long ttl) { +if (nodes.length < 2 || ttl <= 0) { + return; +} +CacheNode parent = ttlCacheTree; +for (int i = 1; i < nodes.length; i++) { + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +parent.addChild(nodes[i], NULL_TTL); +child = parent.getChild(nodes[i]); + } + parent = child; +} +if (parent.ttl == NULL_TTL) { + ttlCount++; +} +parent.ttl = ttl; + } + + /** + * Unset ttl and remove all useless nodes. If the path to be removed is internal node, then just + * reset its ttl. Else, find the sub path and remove it. + * + * @param nodes path to be removed + */ + public void unsetTTL(String[] nodes) { +if (nodes.length < 2) { + return; +} else if (nodes.length == 2) { + // if path equals to root.**, then unset it to configured ttl + if (nodes[0].equals(IoTDBConstant.PATH_ROOT) + && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) { +ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl = +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(); +return; + } +} +CacheNode parent = ttlCacheTree; +int index = 0; +boolean usefulFlag; +CacheNode parentOfSubPathToBeRemoved = null; +for (int i = 1; i < nodes.length; i++) { + usefulFlag = !parent.getChildren().isEmpty() || parent.ttl != NULL_TTL; + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +// there is no matching path on ttl cache tree +return; + } + if (usefulFlag) { +parentOfSubPathToBeRemoved = parent; +index = i; + } + parent = child; +} +// currently, parent is the leaf node of the path to be removed +if (parent.ttl != NULL_TTL) { + ttlCount--; +} + +if (!parent.getChildren().isEmpty()) { + // node to be removed is internal node, then just reset its ttl + parent.ttl = NULL_TTL; + return; +} + +// node to be removed is leaf node, then remove corresponding node of this path from cache tree +if (parentOfSubPathToBeRemoved != null) { + parentOfSubPathToBeRemoved.removeChild(nodes[index]); +} + } + + /** + * Get ttl from cache tree. Return the TTL of the node closest to the path leaf node that has a + * TTL which is not NULL_TTL. + * + * @param nodes should be prefix path or specific device path without wildcard + */ +
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577175083 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL; +ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL); +ttlCount = 1; + } + + /** + * Put ttl into cache tree. + * + * @param nodes should be prefix path or specific device path without wildcard + */ + public void setTTL(String[] nodes, long ttl) { +if (nodes.length < 2 || ttl <= 0) { + return; +} +CacheNode parent = ttlCacheTree; +for (int i = 1; i < nodes.length; i++) { + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +parent.addChild(nodes[i], NULL_TTL); +child = parent.getChild(nodes[i]); + } + parent = child; +} +if (parent.ttl == NULL_TTL) { + ttlCount++; +} +parent.ttl = ttl; + } + + /** + * Unset ttl and remove all useless nodes. If the path to be removed is internal node, then just + * reset its ttl. Else, find the sub path and remove it. + * + * @param nodes path to be removed + */ + public void unsetTTL(String[] nodes) { +if (nodes.length < 2) { + return; +} else if (nodes.length == 2) { + // if path equals to root.**, then unset it to configured ttl + if (nodes[0].equals(IoTDBConstant.PATH_ROOT) + && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) { +ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl = +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(); +return; + } +} +CacheNode parent = ttlCacheTree; +int index = 0; +boolean usefulFlag; Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577172088 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL; +ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL); +ttlCount = 1; + } + + /** + * Put ttl into cache tree. + * + * @param nodes should be prefix path or specific device path without wildcard + */ + public void setTTL(String[] nodes, long ttl) { +if (nodes.length < 2 || ttl <= 0) { + return; +} +CacheNode parent = ttlCacheTree; +for (int i = 1; i < nodes.length; i++) { + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +parent.addChild(nodes[i], NULL_TTL); +child = parent.getChild(nodes[i]); + } + parent = child; +} +if (parent.ttl == NULL_TTL) { + ttlCount++; +} +parent.ttl = ttl; + } + + /** + * Unset ttl and remove all useless nodes. If the path to be removed is internal node, then just + * reset its ttl. Else, find the sub path and remove it. Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577167260 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL; +ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL); +ttlCount = 1; + } + + /** + * Put ttl into cache tree. + * + * @param nodes should be prefix path or specific device path without wildcard + */ + public void setTTL(String[] nodes, long ttl) { +if (nodes.length < 2 || ttl <= 0) { + return; +} +CacheNode parent = ttlCacheTree; +for (int i = 1; i < nodes.length; i++) { + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +parent.addChild(nodes[i], NULL_TTL); +child = parent.getChild(nodes[i]); + } + parent = child; +} +if (parent.ttl == NULL_TTL) { + ttlCount++; +} +parent.ttl = ttl; + } + + /** + * Unset ttl and remove all useless nodes. If the path to be removed is internal node, then just + * reset its ttl. Else, find the sub path and remove it. + * + * @param nodes path to be removed + */ + public void unsetTTL(String[] nodes) { +if (nodes.length < 2) { + return; +} else if (nodes.length == 2) { + // if path equals to root.**, then unset it to configured ttl + if (nodes[0].equals(IoTDBConstant.PATH_ROOT) + && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) { +ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl = +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(); Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Fixed the bug that pure data transfer is forbidden when SchemaRegion is in Simple consensus [iotdb]
SteveYurongSu merged PR #12397: URL: https://github.com/apache/iotdb/pull/12397 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] update Export/ImportDataTestIT [iotdb]
2b3c511 opened a new pull request, #12399: URL: https://github.com/apache/iotdb/pull/12399 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimate of FI memory usage [iotdb]
JackieTien97 commented on code in PR #12393: URL: https://github.com/apache/iotdb/pull/12393#discussion_r1577089745 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java: ## @@ -209,6 +209,8 @@ public enum PlanNodeType { EXPLAIN_ANALYZE((short) 90), PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91), + + FULL_OUTER_TIME_JOIN_REFERENCE((short) 92), Review Comment: what's this for? ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/LocalExecutionPlanContext.java: ## @@ -91,6 +98,10 @@ public class LocalExecutionPlanContext { // use AtomicReference not for thread-safe, just for updating same field in different pipeline private AtomicReference> timePartitions = new AtomicReference<>(); + /** Records the parent of each pipeline. The order of each list does not matter for now. */ + private Map> parentPlanNodeIdToMemoryEstimator = Review Comment: remember to set this map to null, after you check the memory, this Map may occupy huge memory when there are many devices; -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Implement new plan optimizer: AggregationPushDown [iotdb]
liuminghui233 opened a new pull request, #12398: URL: https://github.com/apache/iotdb/pull/12398 see doc https://apache-iotdb.feishu.cn/docx/JiREdjYz3oBjrJxVozXcgs9inS1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Two stage aggregate counter [iotdb]
sonarcloud[bot] commented on PR #12328: URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2072746986 ## [![Quality Gate Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png 'Quality Gate Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) **Quality Gate failed** Failed conditions ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12328=new_coverage=list) (required ≥ 80%) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) (required ≥ A) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) ## ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png '') Catch issues before they fail your Quality Gate with our IDE extension ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png '') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576397733 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL; +ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL); +ttlCount = 1; + } + + /** + * Put ttl into cache tree. + * + * @param nodes should be prefix path or specific device path without wildcard + */ + public void setTTL(String[] nodes, long ttl) { +if (nodes.length < 2 || ttl <= 0) { + return; +} +CacheNode parent = ttlCacheTree; +for (int i = 1; i < nodes.length; i++) { + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +parent.addChild(nodes[i], NULL_TTL); +child = parent.getChild(nodes[i]); + } + parent = child; +} +if (parent.ttl == NULL_TTL) { + ttlCount++; +} +parent.ttl = ttl; + } + + /** + * Unset ttl and remove all useless nodes. If the path to be removed is internal node, then just + * reset its ttl. Else, find the sub path and remove it. + * + * @param nodes path to be removed + */ + public void unsetTTL(String[] nodes) { +if (nodes.length < 2) { + return; +} else if (nodes.length == 2) { + // if path equals to root.**, then unset it to configured ttl + if (nodes[0].equals(IoTDBConstant.PATH_ROOT) + && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) { +ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl = +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(); +return; + } +} +CacheNode parent = ttlCacheTree; +int index = 0; +boolean usefulFlag; +CacheNode parentOfSubPathToBeRemoved = null; +for (int i = 1; i < nodes.length; i++) { + usefulFlag = !parent.getChildren().isEmpty() || parent.ttl != NULL_TTL; + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +// there is no matching path on ttl cache tree +return; + } + if (usefulFlag) { +parentOfSubPathToBeRemoved = parent; +index = i; + } + parent = child; +} +// currently, parent is the leaf node of the path to be removed +if (parent.ttl != NULL_TTL) { + ttlCount--; +} + +if (!parent.getChildren().isEmpty()) { + // node to be removed is internal node, then just reset its ttl + parent.ttl = NULL_TTL; + return; +} + +// node to be removed is leaf node, then remove corresponding node of this path from cache tree +if (parentOfSubPathToBeRemoved != null) { + parentOfSubPathToBeRemoved.removeChild(nodes[index]); +} + } + + /** + * Get ttl from cache tree. Return the TTL of the node closest to the path leaf node that has a + * TTL which is not NULL_TTL. + * + * @param nodes should be prefix path or specific device path without wildcard + */ +
[PR] Pipe: Fixed the bug that pure data transfer is forbidden when SchemaRegion is in Simple consensus [iotdb]
Caideyipi opened a new pull request, #12397: URL: https://github.com/apache/iotdb/pull/12397 ## Description As the title said. This PR has: - [ ] been self-reviewed. - [ ] concurrent read - [ ] concurrent write - [ ] concurrent read and write - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. - [ ] added or updated version, __license__, or notice information - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage. - [ ] added integration tests. - [ ] been tested in a test IoTDB cluster. # Key changed/added classes (or packages if there are too many classes) in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Pipe: Two stage aggregate counter [iotdb]
sonarcloud[bot] commented on PR #12328: URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2072023828 ## [![Quality Gate Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png 'Quality Gate Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) **Quality Gate failed** Failed conditions ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12328=new_coverage=list) (required ≥ 80%) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) (required ≥ A) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) ## ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png '') Catch issues before they fail your Quality Gate with our IDE extension ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png '') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fixed backup data backend running issues [iotdb]
CloudWise-Lukemiao merged PR #12396: URL: https://github.com/apache/iotdb/pull/12396 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix effective ratio calculation method [iotdb]
OneSizeFitsQuorum merged PR #12394: URL: https://github.com/apache/iotdb/pull/12394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576047456 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL; +ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL); +ttlCount = 1; + } + + /** + * Put ttl into cache tree. + * + * @param nodes should be prefix path or specific device path without wildcard + */ + public void setTTL(String[] nodes, long ttl) { +if (nodes.length < 2 || ttl <= 0) { + return; +} +CacheNode parent = ttlCacheTree; +for (int i = 1; i < nodes.length; i++) { + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +parent.addChild(nodes[i], NULL_TTL); +child = parent.getChild(nodes[i]); + } + parent = child; +} +if (parent.ttl == NULL_TTL) { + ttlCount++; +} +parent.ttl = ttl; + } + + /** + * Unset ttl and remove all useless nodes. If the path to be removed is internal node, then just + * reset its ttl. Else, find the sub path and remove it. + * + * @param nodes path to be removed + */ + public void unsetTTL(String[] nodes) { +if (nodes.length < 2) { + return; +} else if (nodes.length == 2) { + // if path equals to root.**, then unset it to configured ttl + if (nodes[0].equals(IoTDBConstant.PATH_ROOT) + && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) { +ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl = +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(); +return; + } +} +CacheNode parent = ttlCacheTree; +int index = 0; +boolean usefulFlag; +CacheNode parentOfSubPathToBeRemoved = null; +for (int i = 1; i < nodes.length; i++) { + usefulFlag = !parent.getChildren().isEmpty() || parent.ttl != NULL_TTL; + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +// there is no matching path on ttl cache tree +return; + } + if (usefulFlag) { +parentOfSubPathToBeRemoved = parent; +index = i; + } + parent = child; +} +// currently, parent is the leaf node of the path to be removed +if (parent.ttl != NULL_TTL) { + ttlCount--; +} + +if (!parent.getChildren().isEmpty()) { + // node to be removed is internal node, then just reset its ttl + parent.ttl = NULL_TTL; + return; +} + +// node to be removed is leaf node, then remove corresponding node of this path from cache tree +if (parentOfSubPathToBeRemoved != null) { + parentOfSubPathToBeRemoved.removeChild(nodes[index]); +} + } + + /** + * Get ttl from cache tree. Return the TTL of the node closest to the path leaf node that has a + * TTL which is not NULL_TTL. + * + * @param nodes should be prefix path or specific device path without wildcard + */ +
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576040934 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL; +ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL); +ttlCount = 1; + } + + /** + * Put ttl into cache tree. + * + * @param nodes should be prefix path or specific device path without wildcard + */ + public void setTTL(String[] nodes, long ttl) { +if (nodes.length < 2 || ttl <= 0) { + return; +} +CacheNode parent = ttlCacheTree; +for (int i = 1; i < nodes.length; i++) { + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +parent.addChild(nodes[i], NULL_TTL); +child = parent.getChild(nodes[i]); + } + parent = child; +} +if (parent.ttl == NULL_TTL) { + ttlCount++; +} +parent.ttl = ttl; + } + + /** + * Unset ttl and remove all useless nodes. If the path to be removed is internal node, then just + * reset its ttl. Else, find the sub path and remove it. + * + * @param nodes path to be removed + */ + public void unsetTTL(String[] nodes) { +if (nodes.length < 2) { + return; +} else if (nodes.length == 2) { + // if path equals to root.**, then unset it to configured ttl + if (nodes[0].equals(IoTDBConstant.PATH_ROOT) + && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) { +ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl = +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(); +return; + } +} +CacheNode parent = ttlCacheTree; Review Comment: Resolved. Rename to `current`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576037933 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL; +ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL); +ttlCount = 1; + } + + /** + * Put ttl into cache tree. + * + * @param nodes should be prefix path or specific device path without wildcard + */ + public void setTTL(String[] nodes, long ttl) { +if (nodes.length < 2 || ttl <= 0) { + return; +} +CacheNode parent = ttlCacheTree; +for (int i = 1; i < nodes.length; i++) { + CacheNode child = parent.getChild(nodes[i]); + if (child == null) { +parent.addChild(nodes[i], NULL_TTL); +child = parent.getChild(nodes[i]); Review Comment: Resolved. `addChild` can return the newly added node ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java: ## @@ -0,0 +1,275 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.schema.ttl; + +import org.apache.iotdb.commons.conf.CommonDescriptor; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils; + +import javax.annotation.concurrent.NotThreadSafe; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.IOException; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +@NotThreadSafe +public class TTLCache { + + private final CacheNode ttlCacheTree; + public static final long NULL_TTL = -1; + + private int ttlCount; + + public TTLCache() { +ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT); +long defaultTTL = +CommonDateTimeUtils.convertMilliTimeWithPrecision( +CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(), + CommonDescriptor.getInstance().getConfig().getTimestampPrecision()); +defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE :
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576029400 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/exception/TTLException.java: ## @@ -0,0 +1,38 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.commons.exception; + +import org.apache.iotdb.commons.conf.CommonDescriptor; + +public class TTLException extends Exception { + + public TTLException(String path) { +super( +String.format( +"Illegal pattern path: %s, pattern path should end with **, otherwise, it should be a specific database or device path without *", +path)); + } + + public TTLException() { +super( +String.format( +"The number of TTL stored in the system has reached threshold %d, please increase the ttl_count parameter.", + CommonDescriptor.getInstance().getConfig().getTTLCountThreshold())); + } +} Review Comment: Resolved, use `TTLIllegalPathException` and `TTLRuleCountNotEnoughException` class instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fixed backup data backend running issues [iotdb]
CloudWise-Lukemiao closed pull request #12395: Fixed backup data backend running issues URL: https://github.com/apache/iotdb/pull/12395 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Some region migration related work [iotdb]
OneSizeFitsQuorum merged PR #12376: URL: https://github.com/apache/iotdb/pull/12376 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576021262 ## iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java: ## @@ -119,6 +119,9 @@ public class CommonConfig { */ private long[] tierTTLInMs = {Long.MAX_VALUE}; + /** The threshold for the number of TTL stored in the system, the default is 1000. */ + private int TTLCountThreshold = 1000; Review Comment: Change to `TTLRuleCount`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576020607 ## iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties: ## @@ -450,6 +450,25 @@ data_replication_factor=1 # Unit: ms # default_ttl_in_ms=-1 +# The threshold for the number of TTL stored in the system, the default is 1000. +# Negative value means the threshold is unlimited. +# Datatype: int +# ttl_count_threshold=1000 + +# The interval of ttl check task in each database. Default is 2 hours. +# Notice: It is not recommended to change it too small, as it will affect the read and write performance of the system. +# Unit: ms +# ttl_check_interval=720 + +# The max expired time of device set with ttl. If the expired time exceeds this value, then the expired data will be cleaned by compaction. Default is 1 month. +# Notice: It is not recommended to change it too small, as it will affect the read and write performance of the system. +# Unit: ms +# max_expired_time=259200 + +# The expired device rate. If the number of expired devices in one tsfile exceeds this value, then expired data of this tsfile will be cleaned by compaction. Review Comment: Resolved. ## iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties: ## @@ -450,6 +450,25 @@ data_replication_factor=1 # Unit: ms # default_ttl_in_ms=-1 +# The threshold for the number of TTL stored in the system, the default is 1000. +# Negative value means the threshold is unlimited. +# Datatype: int +# ttl_count_threshold=1000 + +# The interval of ttl check task in each database. Default is 2 hours. +# Notice: It is not recommended to change it too small, as it will affect the read and write performance of the system. +# Unit: ms +# ttl_check_interval=720 + +# The max expired time of device set with ttl. If the expired time exceeds this value, then the expired data will be cleaned by compaction. Default is 1 month. +# Notice: It is not recommended to change it too small, as it will affect the read and write performance of the system. +# Unit: ms +# max_expired_time=259200 + +# The expired device rate. If the number of expired devices in one tsfile exceeds this value, then expired data of this tsfile will be cleaned by compaction. +# Unit: float Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576019303 ## iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties: ## @@ -450,6 +450,25 @@ data_replication_factor=1 # Unit: ms # default_ttl_in_ms=-1 +# The threshold for the number of TTL stored in the system, the default is 1000. +# Negative value means the threshold is unlimited. +# Datatype: int +# ttl_count_threshold=1000 + +# The interval of ttl check task in each database. Default is 2 hours. +# Notice: It is not recommended to change it too small, as it will affect the read and write performance of the system. +# Unit: ms +# ttl_check_interval=720 + +# The max expired time of device set with ttl. If the expired time exceeds this value, then the expired data will be cleaned by compaction. Default is 1 month. Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576014872 ## iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties: ## @@ -450,6 +450,25 @@ data_replication_factor=1 # Unit: ms # default_ttl_in_ms=-1 +# The threshold for the number of TTL stored in the system, the default is 1000. +# Negative value means the threshold is unlimited. +# Datatype: int +# ttl_count_threshold=1000 + +# The interval of ttl check task in each database. Default is 2 hours. +# Notice: It is not recommended to change it too small, as it will affect the read and write performance of the system. Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576011751 ## iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties: ## @@ -450,6 +450,25 @@ data_replication_factor=1 # Unit: ms # default_ttl_in_ms=-1 +# The threshold for the number of TTL stored in the system, the default is 1000. Review Comment: Resolved. Change to `ttl_rule_count`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576000190 ## iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/settle/SettleCompactionRecoverTest.java: ## @@ -0,0 +1,1137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iotdb.db.storageengine.dataregion.compaction.settle; + +import org.apache.iotdb.commons.exception.IllegalPathException; +import org.apache.iotdb.commons.exception.MetadataException; +import org.apache.iotdb.db.conf.IoTDBDescriptor; +import org.apache.iotdb.db.exception.StorageEngineException; +import org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache; +import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest; +import org.apache.iotdb.db.storageengine.dataregion.compaction.constant.CompactionTaskType; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.impl.FastCompactionPerformer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.subtask.FastCompactionTaskSummary; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.CompactionLogger; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.SimpleCompactionLogger; +import org.apache.iotdb.db.storageengine.dataregion.compaction.utils.CompactionFileGeneratorUtils; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.generator.TsFileNameGenerator; +import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor; +import org.apache.iotdb.tsfile.exception.write.WriteProcessException; +import org.apache.iotdb.tsfile.utils.Pair; + +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.PATH_SEPARATOR; + +public class SettleCompactionRecoverTest extends AbstractCompactionTest { + @Before + public void setUp() + throws IOException, WriteProcessException, MetadataException, InterruptedException { +super.setUp(); +IoTDBDescriptor.getInstance().getConfig().setTargetChunkSize(512); +IoTDBDescriptor.getInstance().getConfig().setTargetChunkPointNum(100); +TSFileDescriptor.getInstance().getConfig().setMaxNumberOfPointsInPage(10); Review Comment: It has been reset in `AbstractCompactionTest.teardown()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575992167 ## iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/TsFileResourceProgressIndexTest.java: ## @@ -277,6 +278,7 @@ public void testProgressIndexMinimumProgressIndexTopologicalSort() { } } + @Ignore Review Comment: I've removed this comment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575990673 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java: ## @@ -0,0 +1,328 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl; + +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.exception.IllegalPathException; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.db.conf.IoTDBConfig; +import org.apache.iotdb.db.conf.IoTDBDescriptor; +import org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils; +import org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector; +import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion; +import org.apache.iotdb.db.storageengine.dataregion.modification.Modification; +import org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex; +import org.apache.iotdb.tsfile.file.metadata.IDeviceID; +import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import static org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED; + +public class SettleSelectorImpl implements ISettleSelector { + private static final Logger LOGGER = + LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME); + private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig(); + + private final boolean heavySelect; + private final String storageGroupName; + private final String dataRegionId; + private final long timePartition; + private final TsFileManager tsFileManager; + private boolean isSeq; + + public SettleSelectorImpl( + boolean heavySelect, + String storageGroupName, + String dataRegionId, + long timePartition, + TsFileManager tsFileManager) { +this.heavySelect = heavySelect; +this.storageGroupName = storageGroupName; +this.dataRegionId = dataRegionId; +this.timePartition = timePartition; +this.tsFileManager = tsFileManager; + } + + static class AllDirtyResource { +List resources = new ArrayList<>(); + +public void add(TsFileResource resource) { + resources.add(resource); +} + +public List getResources() { + return resources; +} + } + + static class PartialDirtyResource { +List resources = new ArrayList<>(); +long totalFileSize = 0; + +public boolean add(TsFileResource resource, long dirtyDataSize) { + resources.add(resource); + totalFileSize += resource.getTsFileSize(); + totalFileSize -= dirtyDataSize; + return checkHasReachedThreshold(); +} + +public List getResources() { + return resources; +} + +public boolean checkHasReachedThreshold() { + return resources.size() >= config.getFileLimitPerInnerTask() + || totalFileSize >= config.getTargetCompactionFileSize(); +} + } + + @Override + public List selectSettleTask(List tsFileResources) { +if
[PR] Fix effective ratio calculation method [iotdb]
HeimingZ opened a new pull request, #12394: URL: https://github.com/apache/iotdb/pull/12394 Use wal node disk size instead of wal manager total size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575929452 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java: ## @@ -0,0 +1,328 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl; + +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.exception.IllegalPathException; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.db.conf.IoTDBConfig; +import org.apache.iotdb.db.conf.IoTDBDescriptor; +import org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils; +import org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector; +import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion; +import org.apache.iotdb.db.storageengine.dataregion.modification.Modification; +import org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex; +import org.apache.iotdb.tsfile.file.metadata.IDeviceID; +import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import static org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED; + +public class SettleSelectorImpl implements ISettleSelector { + private static final Logger LOGGER = + LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME); + private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig(); + + private final boolean heavySelect; + private final String storageGroupName; + private final String dataRegionId; + private final long timePartition; + private final TsFileManager tsFileManager; + private boolean isSeq; + + public SettleSelectorImpl( + boolean heavySelect, + String storageGroupName, + String dataRegionId, + long timePartition, + TsFileManager tsFileManager) { +this.heavySelect = heavySelect; +this.storageGroupName = storageGroupName; +this.dataRegionId = dataRegionId; +this.timePartition = timePartition; +this.tsFileManager = tsFileManager; + } + + static class AllDirtyResource { +List resources = new ArrayList<>(); + +public void add(TsFileResource resource) { + resources.add(resource); +} + +public List getResources() { + return resources; +} + } + + static class PartialDirtyResource { +List resources = new ArrayList<>(); +long totalFileSize = 0; Review Comment: Resolved. `class AllDirtyResource` has been removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575918974 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java: ## @@ -0,0 +1,328 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl; + +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.exception.IllegalPathException; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.db.conf.IoTDBConfig; +import org.apache.iotdb.db.conf.IoTDBDescriptor; +import org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils; +import org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector; +import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion; +import org.apache.iotdb.db.storageengine.dataregion.modification.Modification; +import org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex; +import org.apache.iotdb.tsfile.file.metadata.IDeviceID; +import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import static org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED; + +public class SettleSelectorImpl implements ISettleSelector { + private static final Logger LOGGER = + LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME); + private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig(); + + private final boolean heavySelect; + private final String storageGroupName; + private final String dataRegionId; + private final long timePartition; + private final TsFileManager tsFileManager; + private boolean isSeq; + + public SettleSelectorImpl( + boolean heavySelect, Review Comment: Resolved. I've added comments at line 62. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575911991 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/SettleCompactionTask.java: ## @@ -0,0 +1,389 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task; + +import org.apache.iotdb.db.service.metrics.FileMetrics; +import org.apache.iotdb.db.storageengine.dataregion.compaction.constant.CompactionTaskType; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.exception.CompactionRecoverException; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.CompactionLogAnalyzer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.CompactionLogger; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.SimpleCompactionLogger; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.TsFileIdentifier; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.generator.TsFileNameGenerator; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +/** + * This settle task contains all_deleted files and partial_deleted files. The partial_deleted files + * are divided into several groups, each group may contain one or several files. This task will do + * the following two things respectively: 1. Settle all all_deleted files by deleting them directly. + * 2. Settle partial_deleted files: put the files of each partial_deleted group into an invisible + * innerCompactionTask, and then perform the cleanup work. The source files in a file group will be + * compacted into a target file. + */ +public class SettleCompactionTask extends InnerSpaceCompactionTask { + private List allDeletedFiles; + private double allDeletedFileSize = 0; + private double partialDeletedFileSize = 0; + + private int allDeletedSuccessNum = 0; + + private long totalModsFileSize; + + public SettleCompactionTask( + long timePartition, + TsFileManager tsFileManager, + List allDeletedFiles, + List partialDeletedFiles, Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] iotdb 压缩线程处理异常之后,会无限处理错误数据。手动删除对应的序列也没用 [iotdb]
lintao666 commented on issue #12360: URL: https://github.com/apache/iotdb/issues/12360#issuecomment-2071687177 升级到1.3.1之后,这个压缩路径处理 还是有些问题。路径中包含root 无法压缩,但是对应的时间序列是存在的,也可以查询到数据。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Enhance estimatation of FI memory usage [iotdb]
codecov[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2071684287 ## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/12393?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `96.51163%` with `3 lines` in your changes are missing coverage. Please review. > Project coverage is 43.19%. Comparing base [(`f4804a4`)](https://app.codecov.io/gh/apache/iotdb/commit/f4804a4144076397e254208a150d2762ce716f31?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`1b63668`)](https://app.codecov.io/gh/apache/iotdb/pull/12393?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 44 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/iotdb/pull/12393?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [...ueryengine/plan/planner/LocalExecutionPlanner.java](https://app.codecov.io/gh/apache/iotdb/pull/12393?src=pr=tree=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2FLocalExecutionPlanner.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL0xvY2FsRXhlY3V0aW9uUGxhbm5lci5qYXZh) | 57.14% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/12393?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12393 +/- ## - Coverage 46.40% 43.19% -3.22% Complexity 69 69 Files 3378 3088 -290 Lines207497 190358 -17139 Branches 2494822672-2276 - Hits 9629182221 -14070 + Misses 111206 108137-3069 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/12393?dropdown=coverage=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575827285 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/InnerSpaceCompactionTask.java: ## @@ -283,30 +227,87 @@ protected boolean doCompaction() { String.format("%.2f", costTime), String.format("%.2f", selectedFileSize / 1024.0d / 1024.0d / costTime), summary); - } finally { -Files.deleteIfExists(logFile.toPath()); -// may failed to set status if the status of target resource is DELETED -targetTsFileResource.setStatus(TsFileResourceStatus.NORMAL); } } catch (Exception e) { isSuccess = false; handleException(LOGGER, e); recover(); } finally { releaseAllLocks(); + try { +if (logFile != null) { + Files.deleteIfExists(logFile.toPath()); +} + } catch (IOException e) { +handleException(LOGGER, e); + } + // may fail to set status if the status of target resource is DELETED + targetTsFileResource.setStatus(TsFileResourceStatus.NORMAL); } return isSuccess; } + protected void compact(SimpleCompactionLogger compactionLogger) throws Exception { +// carry out the compaction +targetTsFileList = new ArrayList<>(Collections.singletonList(targetTsFileResource)); +performer.setSourceFiles(selectedTsFileResourceList); +// As elements in targetFiles may be removed in performer, we should use a mutable list +// instead of Collections.singletonList() +performer.setTargetFiles(targetTsFileList); +performer.setSummary(summary); +performer.perform(); + +prepareTargetFiles(); + +if (Thread.currentThread().isInterrupted() || summary.isCancel()) { + throw new InterruptedException( + String.format("%s-%s [Compaction] abort", storageGroupName, dataRegionId)); +} + +validateCompactionResult( +sequence ? selectedTsFileResourceList : Collections.emptyList(), +sequence ? Collections.emptyList() : selectedTsFileResourceList, +targetTsFileList); + +// replace the old files with new file, the new is in same position as the old +tsFileManager.replace( +sequence ? selectedTsFileResourceList : Collections.emptyList(), +sequence ? Collections.emptyList() : selectedTsFileResourceList, +targetTsFileList, +timePartition); + +// release the read lock of all source files, and get the write lock of them to delete them +for (int i = 0; i < selectedTsFileResourceList.size(); ++i) { + selectedTsFileResourceList.get(i).writeLock(); + isHoldingWriteLock[i] = true; +} Review Comment: Comments have been updated. This is old code, compaction now no longer requires adding and releasing read locks of source files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575818839 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/cache/schema/DataNodeTTLCache.java: ## @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.queryengine.plan.analyze.cache.schema; + +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.schema.ttl.TTLCache; +import org.apache.iotdb.commons.utils.TestOnly; + +import java.util.Map; +import java.util.concurrent.locks.ReentrantReadWriteLock; + +public class DataNodeTTLCache { + private final TTLCache ttlCache; + + private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(); + + private DataNodeTTLCache() { +ttlCache = new TTLCache(); + } + + public static DataNodeTTLCache getInstance() { +return DataNodeTTLCacheHolder.INSTANCE; + } + + private static class DataNodeTTLCacheHolder { +private static final DataNodeTTLCache INSTANCE = new DataNodeTTLCache(); + } + + public void setTTL(String path, long ttl) { +lock.writeLock().lock(); +try { + ttlCache.setTTL(path.split("\\" + IoTDBConstant.PATH_SEPARATOR), ttl); Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575815531 ## iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/TTLInfo.java: ## @@ -0,0 +1,166 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.confignode.persistence; + +import org.apache.iotdb.common.rpc.thrift.TSStatus; +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.schema.ttl.TTLCache; +import org.apache.iotdb.commons.snapshot.SnapshotProcessor; +import org.apache.iotdb.confignode.consensus.request.write.database.SetTTLPlan; +import org.apache.iotdb.confignode.consensus.response.ttl.ShowTTLResp; +import org.apache.iotdb.rpc.RpcUtils; +import org.apache.iotdb.rpc.TSStatusCode; + +import org.apache.thrift.TException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.util.Map; +import java.util.concurrent.locks.ReadWriteLock; +import java.util.concurrent.locks.ReentrantReadWriteLock; + +public class TTLInfo implements SnapshotProcessor { + private static final String SNAPSHOT_FILENAME = "ttl_info.bin"; + private static final Logger LOGGER = LoggerFactory.getLogger(TTLInfo.class); + + private final TTLCache ttlCache; + + private final ReadWriteLock lock; + + public TTLInfo() { +ttlCache = new TTLCache(); +lock = new ReentrantReadWriteLock(); + } + + public TSStatus setTTL(SetTTLPlan plan) { +lock.writeLock().lock(); +try { + ttlCache.setTTL(plan.getPathPattern(), plan.getTTL()); + if (plan.isDataBase()) { +// set ttl to path.** +ttlCache.setTTL( +new PartialPath(plan.getPathPattern()) +.concatNode(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD) +.getNodes(), +plan.getTTL()); + } +} finally { + lock.writeLock().unlock(); +} +return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS); + } + + public TSStatus unsetTTL(SetTTLPlan plan) { +lock.writeLock().lock(); +try { + ttlCache.unsetTTL(plan.getPathPattern()); + if (plan.isDataBase()) { +// unset ttl to path.** +ttlCache.unsetTTL( +new PartialPath(plan.getPathPattern()) +.concatNode(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD) +.getNodes()); + } +} finally { + lock.writeLock().unlock(); +} +return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS); + } + + public ShowTTLResp showAllTTL() { +lock.readLock().lock(); +ShowTTLResp resp = new ShowTTLResp(); +try { + Map pathTTLMap = ttlCache.getAllPathTTL(); + resp.setPathTTLMap(pathTTLMap); + resp.setStatus(new TSStatus(TSStatusCode.SUCCESS_STATUS.getStatusCode())); +} finally { + lock.readLock().unlock(); +} +return resp; + } Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [rel/1.3] Fixed rest insertTablet http api lost data event [iotdb]
CloudWise-Lukemiao commented on PR #12386: URL: https://github.com/apache/iotdb/pull/12386#issuecomment-2071648982 http://127.0.0.1:18080/rest/v2/insertTablet When inserting with the following parameters, when there are duplicates in the timestamp and more than half of the values in the values are null {"is_aligned":false,"device":"root.mqtt.topic.abc454","data_types":["TEXT","INT32","FLOAT"],"measurements":["foo","temp","weight"],"timestamps":[1713495895809,1713499707255,1713495893809,1713495892809,1713495891809,1713495890809,1713495889809,171349509,1713495887809,1713495893809,1713495892809,1713495891809,1713495890809,1713495889809,171349509],"values":[["bar",null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,41,32,33,34,39,38,37,36,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,87.0,87.3,87.3,87.0,87.3,87.3]]} ![image](https://github.com/apache/iotdb/assets/76942485/d6c24e27-e65c-4623-a01e-0382088758b0) After repair, as shown in the following figure: ![image](https://github.com/apache/iotdb/assets/76942485/ab2a1f10-c175-46a5-8879-ca28c1da0b5b) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575802400 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java: ## @@ -0,0 +1,328 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl; + +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.exception.IllegalPathException; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.db.conf.IoTDBConfig; +import org.apache.iotdb.db.conf.IoTDBDescriptor; +import org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils; +import org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector; +import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion; +import org.apache.iotdb.db.storageengine.dataregion.modification.Modification; +import org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex; +import org.apache.iotdb.tsfile.file.metadata.IDeviceID; +import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import static org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED; + +public class SettleSelectorImpl implements ISettleSelector { + private static final Logger LOGGER = + LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME); + private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig(); + + private final boolean heavySelect; + private final String storageGroupName; + private final String dataRegionId; + private final long timePartition; + private final TsFileManager tsFileManager; + private boolean isSeq; + + public SettleSelectorImpl( + boolean heavySelect, + String storageGroupName, + String dataRegionId, + long timePartition, + TsFileManager tsFileManager) { +this.heavySelect = heavySelect; +this.storageGroupName = storageGroupName; +this.dataRegionId = dataRegionId; +this.timePartition = timePartition; +this.tsFileManager = tsFileManager; + } + + static class AllDirtyResource { +List resources = new ArrayList<>(); + +public void add(TsFileResource resource) { + resources.add(resource); +} + +public List getResources() { + return resources; +} + } + + static class PartialDirtyResource { +List resources = new ArrayList<>(); +long totalFileSize = 0; + +public boolean add(TsFileResource resource, long dirtyDataSize) { + resources.add(resource); + totalFileSize += resource.getTsFileSize(); + totalFileSize -= dirtyDataSize; + return checkHasReachedThreshold(); +} + +public List getResources() { + return resources; +} + +public boolean checkHasReachedThreshold() { + return resources.size() >= config.getFileLimitPerInnerTask() + || totalFileSize >= config.getTargetCompactionFileSize(); +} + } + + @Override + public List selectSettleTask(List tsFileResources) { +if
Re: [PR] Enhance estimatation of FI memory usage [iotdb]
sonarcloud[bot] commented on PR #12393: URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2071616292 ## [![Quality Gate Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png 'Quality Gate Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) **Quality Gate failed** Failed conditions ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list) (required ≥ 80%) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575781065 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java: ## @@ -0,0 +1,328 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl; + +import org.apache.iotdb.commons.conf.IoTDBConstant; +import org.apache.iotdb.commons.exception.IllegalPathException; +import org.apache.iotdb.commons.path.PartialPath; +import org.apache.iotdb.commons.utils.CommonDateTimeUtils; +import org.apache.iotdb.db.conf.IoTDBConfig; +import org.apache.iotdb.db.conf.IoTDBDescriptor; +import org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask; +import org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils; +import org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector; +import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion; +import org.apache.iotdb.db.storageengine.dataregion.modification.Modification; +import org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex; +import org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex; +import org.apache.iotdb.tsfile.file.metadata.IDeviceID; +import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +import static org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED; + +public class SettleSelectorImpl implements ISettleSelector { + private static final Logger LOGGER = + LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME); + private static final IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig(); + + private final boolean heavySelect; + private final String storageGroupName; + private final String dataRegionId; + private final long timePartition; + private final TsFileManager tsFileManager; + private boolean isSeq; + + public SettleSelectorImpl( + boolean heavySelect, + String storageGroupName, + String dataRegionId, + long timePartition, + TsFileManager tsFileManager) { +this.heavySelect = heavySelect; +this.storageGroupName = storageGroupName; +this.dataRegionId = dataRegionId; +this.timePartition = timePartition; +this.tsFileManager = tsFileManager; + } + + static class AllDirtyResource { +List resources = new ArrayList<>(); + +public void add(TsFileResource resource) { + resources.add(resource); +} + +public List getResources() { + return resources; +} + } + + static class PartialDirtyResource { +List resources = new ArrayList<>(); +long totalFileSize = 0; + +public boolean add(TsFileResource resource, long dirtyDataSize) { + resources.add(resource); + totalFileSize += resource.getTsFileSize(); + totalFileSize -= dirtyDataSize; + return checkHasReachedThreshold(); +} + +public List getResources() { + return resources; +} + +public boolean checkHasReachedThreshold() { + return resources.size() >= config.getFileLimitPerInnerTask() + || totalFileSize >= config.getTargetCompactionFileSize(); +} + } + + @Override + public List selectSettleTask(List tsFileResources) { +if
Re: [PR] Pipe: Two stage aggregate counter [iotdb]
sonarcloud[bot] commented on PR #12328: URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2071612225 ## [![Quality Gate Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png 'Quality Gate Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) **Quality Gate failed** Failed conditions ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12328=new_coverage=list) (required ≥ 80%) ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png '') [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) (required ≥ A) [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) ## ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png '') Catch issues before they fail your Quality Gate with our IDE extension ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png '') [SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Enhance estimatation of FI memory usage [iotdb]
lancelly opened a new pull request, #12393: URL: https://github.com/apache/iotdb/pull/12393 Pipelines can be organized into a tree structure. This PR is used to estimate the memory usage during the execution of a pipeline. Currently, it is assumed that in the most extreme cases, the child pipelines under a ConsumeAllChildren type pipeline will all run simultaneously. Therefore, when considering memory usage, all children must be accounted for. For ConsumeChildrenOneByOne type pipelines, only some of the child pipelines will run at the same time. Therefore, only the memory usage of these active children needs to be considered, along with the static memory occupied by the inactive children. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] feature add iotdb backup [iotdb]
HTHou merged PR #12365: URL: https://github.com/apache/iotdb/pull/12365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] fix: 下载换行样式 [iotdb-docs]
qiaojialin merged PR #225: URL: https://github.com/apache/iotdb-docs/pull/225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fixed rest service insertTablet lost data event return SUCCESS_STATUS [iotdb]
CloudWise-Lukemiao commented on PR #12387: URL: https://github.com/apache/iotdb/pull/12387#issuecomment-2071548937 http://127.0.0.1:18080/rest/v2/insertTablet When inserting with the following parameters, when there are duplicates in the timestamp and more than half of the values in the values are null {"is_aligned":false,"device":"root.mqtt.topic.abc454","data_types":["TEXT","INT32","FLOAT"],"measurements":["foo","temp","weight"],"timestamps":[1713495895809,1713499707255,1713495893809,1713495892809,1713495891809,1713495890809,1713495889809,171349509,1713495887809,1713495893809,1713495892809,1713495891809,1713495890809,1713495889809,171349509],"values":[["bar",null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,41,32,33,34,39,38,37,36,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,87.0,87.3,87.3,87.0,87.3,87.3]]} After repair, as shown in the following figure: https://github.com/apache/iotdb/assets/76942485/e9688829-cbca-4d2f-9903-c7487bf39aa6;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Provide more precise info for exception during conf loading [iotdb]
OneSizeFitsQuorum merged PR #12382: URL: https://github.com/apache/iotdb/pull/12382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] fix: 下载换行样式 [iotdb-docs]
tongshs opened a new pull request, #225: URL: https://github.com/apache/iotdb-docs/pull/225 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] insertTablet http api lost data event return SUCCESS_STATUS [iotdb]
CritasWang commented on issue #12375: URL: https://github.com/apache/iotdb/issues/12375#issuecomment-2071513726 https://github.com/apache/iotdb/pull/12387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575714841 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/schedule/CompactionScheduleTaskManager.java: ## Review Comment: This is reasonable, equivalent to stopping all compaction selector threads and TTL checker threads when starting to repair data, and then resuming them after the repair is complete. Moreover, users are unaware of the number of threads involved in the repair. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
choubenson commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575710663 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/memory/SystemInfo.java: ## Review Comment: Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Implementation of device TTL [iotdb]
shuwenwei commented on code in PR #12122: URL: https://github.com/apache/iotdb/pull/12122#discussion_r1565391622 ## iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/memory/SystemInfo.java: ## Review Comment: Method `addCompactionMemoryCost( CompactionTaskType taskType, long memoryCost, long timeOutInSecond)` is no longer used after a past change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org