[GitHub] [incubator-iotdb] LebronAl edited a comment on pull request #1691: [IOTDB-814] Cache and redirect distributed IoTDB's leader for write requests in IoTDB's client
LebronAl edited a comment on pull request #1691: URL: https://github.com/apache/incubator-iotdb/pull/1691#issuecomment-689985530 # Benchmark Parameter CLIENT_NUMBER=1 GROUP_NUMBER=1 DEVICE_NUMBER=20 SENSOR_NUMBER=20 BATCH_SIZE=50 LOOP=1000 DATA_TYPE=DOUBLE INSERT_MODE=session # Cluster config default_replica_num=1 enablePartition=false two nodes's cluster(34,35) # Cluster_new's latest Version ## forward(benchmark writes data to 34, sg belongs to 35,internal forward)  ## non-forward(benchmark writes data to 35, sg belongs to 35,no need to forward)  # Cluster_new_cache_leader's latest Version ## Close cache leader ### forward(benchmark writes data to 34, sg belongs to 35,internal forward)  ### non-forward(benchmark writes data to 35, sg belongs to 35,no need to forward)  ## Open cache leader ### little forward(benchmark writes data to 34, sg belongs to 35,internal forward first write for each deviceId + cache leader)  ### non-forward(benchmark writes data to 35, sg belongs to 35,no need to forward)  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl edited a comment on pull request #1691: [IOTDB-814] Cache and redirect distributed IoTDB's leader for write requests in IoTDB's client
LebronAl edited a comment on pull request #1691: URL: https://github.com/apache/incubator-iotdb/pull/1691#issuecomment-689985530 # Benchmark Parameter CLIENT_NUMBER=1 GROUP_NUMBER=1 DEVICE_NUMBER=20 SENSOR_NUMBER=20 BATCH_SIZE=50 LOOP=1000 DATA_TYPE=DOUBLE INSERT_MODE=session # Cluster config default_replica_num=1 enablePartition=false two nodes's cluster(34,35) # Cluster_new's latest Version ## forward(benchmark writes data to 34, sg belongs to 35,internal forward)  ## non-forward(benchmark writes data to 35, sg belongs to 35,no need to forward)  # Cluster_new_cache_leader's latest Version ## Close cache leader ### forward(benchmark writes data to 34, sg belongs to 35,internal forward)  ### non-forward(benchmark writes data to 35, sg belongs to 35,no need to forward)  ## Open cache leader ### little forward(benchmark writes data to 34, sg belongs to 35,internal forward first write for each deviceId + cache leader)  ### non-forward(benchmark writes data to 34, sg belongs to 35,no need to forward)  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl commented on pull request #1691: [IOTDB-814] Cache and redirect distributed IoTDB's leader for write requests in IoTDB's client
LebronAl commented on pull request #1691: URL: https://github.com/apache/incubator-iotdb/pull/1691#issuecomment-689985530 # Benchmark Parameter CLIENT_NUMBER=1 GROUP_NUMBER=1 DEVICE_NUMBER=20 SENSOR_NUMBER=20 BATCH_SIZE=50 LOOP=1000 DATA_TYPE=DOUBLE INSERT_MODE=session # Cluster config default_replica_num=1 enablePartition=false two nodes's cluster(34,35) # Cluster_new's latest Version ## forward(benchmark writes data to 34, sg belongs to 35,internal forward)  ## non-forward(benchmark writes data to 35, sg belongs to 35,no need to forward)  # Cluster_new_cache_leader's latest Version ## Close cache leader ### forward(benchmark writes data to 34, sg belongs to 35,internal forward)  ### non-forward(benchmark writes data to 35, sg belongs to 35,no need to forward)  ## Open cache leader ### little forward(benchmark writes data to 34, sg belongs to 35,internal forward first write for each deviceId + cache leader)  ### non-fowward(benchmark writes data to 34, sg belongs to 35,no need to forward)  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #460: [IOTDB-68] New shared-nothing cluster
sonarcloud[bot] removed a comment on pull request #460: URL: https://github.com/apache/incubator-iotdb/pull/460#issuecomment-689442314 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/issue s?id=apache_incubator-iotdb=460=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [82 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460=new_duplicated_lines_density=list) [1.1% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #460: [IOTDB-68] New shared-nothing cluster
sonarcloud[bot] commented on pull request #460: URL: https://github.com/apache/incubator-iotdb/pull/460#issuecomment-689981742 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/issue s?id=apache_incubator-iotdb=460=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [83 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460=new_duplicated_lines_density=list) [1.1% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1650: [IOTDB-848] support order by time asc/desc
sonarcloud[bot] removed a comment on pull request #1650: URL: https://github.com/apache/incubator-iotdb/pull/1650#issuecomment-689521389 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/i ssues?id=apache_incubator-iotdb=1650=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [18 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650=new_duplicated_lines_density=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1650: [IOTDB-848] support order by time asc/desc
sonarcloud[bot] commented on pull request #1650: URL: https://github.com/apache/incubator-iotdb/pull/1650#issuecomment-689958738 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/i ssues?id=apache_incubator-iotdb=1650=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [20 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650=new_duplicated_lines_density=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] ATM006 opened a new issue #1719: IoTDB是否支持自动重连功能
ATM006 opened a new issue #1719: URL: https://github.com/apache/incubator-iotdb/issues/1719 **Is your feature request related to a problem? Please describe.** A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] **Describe the solution you'd like** A clear and concise description of what you want to happen. **Describe alternatives you've considered** A clear and concise description of any alternative solutions or features you've considered. **Additional context** Add any other context or screenshots about the feature request here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] ATM006 opened a new issue #1718: iotdb0.10.0报错如何解决
ATM006 opened a new issue #1718: URL: https://github.com/apache/incubator-iotdb/issues/1718  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #1650: [IOTDB-848] support order by time asc/desc
liutaohua commented on a change in pull request #1650: URL: https://github.com/apache/incubator-iotdb/pull/1650#discussion_r486036920 ## File path: docs/zh/UserGuide/Operation Manual/DML Data Manipulation Language.md ## @@ -156,6 +156,14 @@ select wf01.wt01.status,wf02.wt02.hardware from root.ln where (time > 2017-11-01 该SQL语句的执行结果如下： https://user-images.githubusercontent.com/13203019/51577450-dcfe0800-1ef4-11e9-9399-4ba2b2b7fb73.jpg;> + 根据时间降序返回 +IoTDB 在 0.10.x 之后支持 'order by time' 语句, 用于对结果按照时间进行降序展示。例如，SQL语句为： +```sql +select * from root.ln where time > 1 order by time desc limit 1; Review comment: 是的，我修改为了 `limit 10` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #1650: [IOTDB-848] support order by time asc/desc
liutaohua commented on a change in pull request #1650:
URL: https://github.com/apache/incubator-iotdb/pull/1650#discussion_r486036824
##
File path: docs/zh/SystemDesign/DataQuery/OrderByTimeQuery.md
##
@@ -0,0 +1,181 @@
+
+
+# 按时间倒序查询
+
+## 实现原理
+
+倒序分为 3 个维度来实现，由内核层到用户层分别为: `TsFile`、`BatchData`、`数据集`.
+1. `TsFile` 是用来存储所有原始数据的文件格式，在存储时分为 `顺序文件` 和 `乱序文件`.
+`ChunkData` 是 `TsFile` 中的基本数据块，保存了具体某一个测点的数据，且**数据是按照时间升序存储的**.
+2. `BatchData` 是基础数据交换结构，无论是从文件系统还是缓存读出的数据都会被转换成 `BatchData` 结构.
+3. `数据集` 是封装根据用户输入的 `SQL` 语句的结果集格式，转换和封装 `BatchData` 中的数据。
+
+实际系统需要考虑的方面会更多，但主要为以上 3 点。
+下面分别介绍各个层面如何实现基于时间的倒序查询:
+
+## TsFile
+1.`TsFile` 在文件末尾存储了各个测点在文件中的最小时间(开始时间)和最大时间(结束时间)，
+所以当我们按照开始时间升序排列文件并读取数据时,是升序读取;当使用开始时间倒序排列文件并读取数据时,
+是**伪倒序读取**(因为文件内依然是升序存储的),数据示例:
+
+```
+按照开始时间升序排列 (1,6)
+ +---TsFile1---+ +---TsFile2---+
+ | 1,2,3,4,5 | | 6,7,8,9,10 |
+ +-+ +-+
+
+按照开始时间降序排列 (10,5)
+ +---TsFile2---+ +---TsFile1---+
+ | 6,7,8,9,10 | | 1,2,3,4,5 |
+ +-+ +-+
+```
+
+2.上面的示例只描述了顺序文件，乱序文件的特点是: 文件中存储的数据与其他文件的数据存在相交，
+不仅仅和顺序文件相交，乱序和乱序文件之间也可能相交。
+
+```
+ +---TsFile1---+ +---TsFile2---+
+ | 1,2,3,4,5 | | 6,7,8,9,10 |
+ +-+ +-+
+ +--unseq1---+
+ | 2,11 |
+ +---+
+ +-unseq2-+
+ | 3,4,5 |
+ ++
+
+```
+
+3.对于**相交**的部分，需要使用`多路归并`查询出最终结果。具体处理细节可以参见[SeriesReader](../DataQuery/SeriesReader.md).
+
+4.对于文件层而言，我们抽象出了`TimeOrderUtils`，用来完成对升序、降序不同实现的方法提取。
+
+```java
+ public interface TimeOrderUtils {
+//排序使用的字段，升序使用开始时间，降序使用结束时间。
+//TsFileResource面对文件相关，Statistics面向Chunk和Page相关
+long getOrderTime(Statistics statistics);
+long getOrderTime(TsFileResource fileResource);
+
+//判断是否相交使用的时间依据，升序使用结束时间，降序使用开始时间
+long getOverlapCheckTime(Statistics range);
+
+//根据提供的时间判断是否相交，left为依据时间，right为要判断的文件
+boolean isOverlapped(Statistics left, Statistics right);
+boolean isOverlapped(long time, Statistics right);
+boolean isOverlapped(long time, TsFileResource right);
+
+//对于有序文件的遍历规则，升序时正序遍历，降序时倒序遍历
+TsFileResource getNextSeqFileResource(List seqResources,
boolean isDelete);
+
+//对于所有文件层的临时缓存排序规则的实现
+ Comparator comparingLong(ToLongFunction keyExtractor);
+
+//循环读取数据时判断是否到达了当前设置的停止点
+boolean isExcessEndpoint(long time, long endpointTime);
+
+//或许当前使用的排序方式
+boolean getAscending();
+ }
+```
+
+5.完成了上述步骤，倒序取出来的数据如下：
+
+```
+6,7,8,9,10,1,2,3,4,5
+```
+
+这是因为数据在Chunk、Page、缓存中都是升序存储，所以转移到 `BatchData` 中查看就是示例中的样子。
+
+## BatchData
+1.为了兼容原有的文件读取数据方式，所以在数据放到 `BatchData` 之后倒序读取集合里的数据就可以得到最终结果。
+
+BatchData类似一个二维数组的格式，封装了写入和读取的方法，对于倒序，抽象了下面的子类：
+```java
+
+public class DescBatchData extends BatchData {
+ //判断是否还有值，正序时候判断是否读到写入的size位置，倒序就是判断是否读到了0的位置
+ public boolean hasCurrent();
+ //指针移动到下一个数据，正序为 index+1 操作， 倒序为 index-1 操作
+ public void next();
+ //重制指针位置，正序为重制到0，倒序重制到数据写入位置
+ public void resetBatchData();
+ //就像是Nio中的Buffer类的flip操作，在数据写入完成时做的操作，为读取做好准备
+ public BatchData flip();
+ //使用具体时间获取值，正序时index从0到size遍历，倒序时从size到0遍历
+ public Object getValueInTimestamp(long time);
+}
+```
+
+2.在需要倒序处理数据的位置构建`DescBatchData`类，为此，特意设计了一个`BatchDataFactory`类，
+根据ascending参数判断生产哪个类型的BatchData。
+
+3.至此数据从磁盘和缓存中取出的数据都是严格按照时间排序的了。
+
+## 数据集
+1.原始数据查询
+```sql
+select * from root order by time desc
+select * from root where root.ln.d1.s1>100 order by time desc
+select * from root where time >100 order by time desc
+```
+原始数据查询中，只要向`SeriesReader`传递了正确的ascending参数就能得到正确的结果。
+
+2.GroupBy
+```sql
+select max_time(*) from root group by ((0,10],2ms) order by time desc
+```
+如上所示的Sql中,groupBy查询的区间分别为:
+```
+[0-2),[2-4),[4-6),[6-8),[8-10)
+```
+倒序查询只需要将时间计算的过程修改为:
+```
+[8-10),[6-8),[4-6),[2-4),[0-2)
+```
+
+>计算方法为:
+>
+> ((结束时间-开始时间)/步长值） 结果向上取整，得到的值为GroupBy中会发生的遍历次数(记为i)。
+>
+> (步长 * (i-1) + 开始时间) 结果为第i次遍历的开始时间值。
+
+
+对于BatchData中的数据大于每次遍历的结束时间时，进行skip，然后对 `AggregateResult` 新增
+minBound计算限制,限制为第次遍历的开始时间值，
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] ATM006 commented on issue #1640: IoTDB是否支持ORDER排序功能
ATM006 commented on issue #1640: URL: https://github.com/apache/incubator-iotdb/issues/1640#issuecomment-689943883 请问当前0.10.1版本支持了吗 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #1650: [IOTDB-848] support order by time asc/desc
liutaohua commented on a change in pull request #1650:
URL: https://github.com/apache/incubator-iotdb/pull/1650#discussion_r486031065
##
File path:
server/src/main/java/org/apache/iotdb/db/query/dataset/groupby/GroupByWithoutValueFilterDataSet.java
##
@@ -140,9 +142,37 @@ protected RowRecord nextWithoutConstraint() throws
IOException {
return record;
}
- protected GroupByExecutor getGroupByExecutor(PartialPath path, Set
allSensors, TSDataType dataType,
- QueryContext context, Filter
timeFilter, TsFileFilter fileFilter)
- throws StorageEngineException, QueryProcessException {
-return new LocalGroupByExecutor(path, allSensors, dataType, context,
timeFilter, fileFilter);
+ @Override
+ public Pair peekNextNotNullValue(Path path, int i) throws
IOException {
+Pair result = null;
+long nextStartTime = curStartTime;
+long nextEndTime;
+do {
+ if (ascending) {
+nextStartTime += plan.getSlidingStep();
+if (nextStartTime < plan.getEndTime()) {
+ nextEndTime = Math.min(nextStartTime + plan.getInterval(),
plan.getEndTime());
+} else {
+ return null;
+}
+ } else {
Review comment:
the code has been removed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #1650: [IOTDB-848] support order by time asc/desc
liutaohua commented on a change in pull request #1650:
URL: https://github.com/apache/incubator-iotdb/pull/1650#discussion_r486030810
##
File path:
server/src/main/java/org/apache/iotdb/db/query/dataset/groupby/GroupByWithValueFilterDataSet.java
##
@@ -160,6 +171,24 @@ protected RowRecord nextWithoutConstraint() throws
IOException {
return constructRowRecord(aggregateResultList);
}
+ @Override
+ public Pair peekNextNotNullValue(Path path, int i) throws
IOException {
+long[] timestampArray = new long[1];
+AggregateResult aggrResultByName =
AggregateResultFactory.getAggrResultByName(
+groupByTimePlan.getDeduplicatedAggregations().get(i),
+groupByTimePlan.getDeduplicatedDataTypes().get(i));
+
+int timeArrayLength = 0;
+if (hasCachedTimestamp) {
+ timestampArray[timeArrayLength++] = timestamp;
+
+ aggrResultByName
+ .updateResultUsingTimestamps(timestampArray, timeArrayLength,
allDataReaderList.get(i));
+ return new Pair<>(timestamp, aggrResultByName.getResult());
+}
+return null;
Review comment:
thx,fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] liutaohua commented on a change in pull request #1650: [IOTDB-848] support order by time asc/desc
liutaohua commented on a change in pull request #1650:
URL: https://github.com/apache/incubator-iotdb/pull/1650#discussion_r486030444
##
File path:
server/src/main/java/org/apache/iotdb/db/query/dataset/groupby/GroupByEngineDataSet.java
##
@@ -31,18 +32,13 @@
public abstract class GroupByEngineDataSet extends QueryDataSet {
protected long queryId;
- protected long interval;
- protected long slidingStep;
- // total query [startTime, endTime)
- protected long startTime;
- protected long endTime;
+ protected GroupByTimePlan plan;
Review comment:
u're right. It's been rolled back
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] yhwang-hbl commented on a change in pull request #1705: [IOTDB-873] Add count devices DDL
yhwang-hbl commented on a change in pull request #1705:
URL: https://github.com/apache/incubator-iotdb/pull/1705#discussion_r486019410
##
File path: server/src/main/java/org/apache/iotdb/db/metadata/MTree.java
##
@@ -695,6 +708,38 @@ private int getCount(MNode node, String[] nodes, int idx)
throws MetadataExcepti
}
}
+ /**
+ * Traverse the MTree to get the count of devices.
+ */
+ private int getDevicesCount(MNode node, String[] nodes, int idx) throws
MetadataException {
+String nodeReg = MetaUtils.getNodeRegByIdx(idx, nodes);
+int cnt = 0;
+if (!(PATH_WILDCARD).equals(nodeReg)) {
+ if (node.hasChild(nodeReg)) {
+if (node.getChild(nodeReg) instanceof MeasurementMNode && idx >=
nodes.length) {
+ cnt++;
+} else {
+ cnt += getDevicesCount(node.getChild(nodeReg), nodes, idx + 1);
+}
+ }
+} else {
+ boolean deviceAdded = false;
+ for (MNode child : node.getChildren().values()) {
+if (child instanceof MeasurementMNode) {
+ if (!deviceAdded) {
+cnt++;
+deviceAdded = true;
+ } else {
+break;
+ }
+} else {
+ cnt += getDevicesCount(child, nodes, idx + 1);
+}
+ }
Review comment:
> Ah.. good point. Maybe @yhwang-hbl could add a specific test case for
this example ; )
OK，i will improve it and add a test for the corresponding situation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] yhwang-hbl commented on a change in pull request #1705: [IOTDB-873] Add count devices DDL
yhwang-hbl commented on a change in pull request #1705:
URL: https://github.com/apache/incubator-iotdb/pull/1705#discussion_r486019128
##
File path: server/src/main/java/org/apache/iotdb/db/metadata/MTree.java
##
@@ -695,6 +708,38 @@ private int getCount(MNode node, String[] nodes, int idx)
throws MetadataExcepti
}
}
+ /**
+ * Traverse the MTree to get the count of devices.
+ */
+ private int getDevicesCount(MNode node, String[] nodes, int idx) throws
MetadataException {
+String nodeReg = MetaUtils.getNodeRegByIdx(idx, nodes);
+int cnt = 0;
+if (!(PATH_WILDCARD).equals(nodeReg)) {
+ if (node.hasChild(nodeReg)) {
+if (node.getChild(nodeReg) instanceof MeasurementMNode && idx >=
nodes.length) {
+ cnt++;
+} else {
+ cnt += getDevicesCount(node.getChild(nodeReg), nodes, idx + 1);
+}
+ }
+} else {
+ boolean deviceAdded = false;
+ for (MNode child : node.getChildren().values()) {
+if (child instanceof MeasurementMNode) {
+ if (!deviceAdded) {
+cnt++;
+deviceAdded = true;
+ } else {
+break;
+ }
+} else {
+ cnt += getDevicesCount(child, nodes, idx + 1);
+}
+ }
Review comment:
> Hi, we support subdevice like "root.sg.device1.subdevice1.sensor1"
while subdevice1 is a measurementNode and a device. In your case, the
subdevice1 will be ignored.
>
> You can refer to the Mtree.findDevice() method.
hi, I didn't know the concept of sub devices before. I did a self-test on my
code, which will filter out the situation you said. I will improve it,thanks
for your guidance
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] LebronAl commented on pull request #1713: Python method "set_time_zone" exception message
LebronAl commented on pull request #1713: URL: https://github.com/apache/incubator-iotdb/pull/1713#issuecomment-689913410 [Python sessionUT](https://github.com/apache/incubator-iotdb/pull/1578) is still reviewing. You can take a review~ Thanks~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #1705: [IOTDB-873] Add count devices DDL
samperson1997 commented on a change in pull request #1705:
URL: https://github.com/apache/incubator-iotdb/pull/1705#discussion_r485630160
##
File path: server/src/main/java/org/apache/iotdb/db/metadata/MTree.java
##
@@ -695,6 +708,38 @@ private int getCount(MNode node, String[] nodes, int idx)
throws MetadataExcepti
}
}
+ /**
+ * Traverse the MTree to get the count of devices.
+ */
+ private int getDevicesCount(MNode node, String[] nodes, int idx) throws
MetadataException {
+String nodeReg = MetaUtils.getNodeRegByIdx(idx, nodes);
+int cnt = 0;
+if (!(PATH_WILDCARD).equals(nodeReg)) {
+ if (node.hasChild(nodeReg)) {
+if (node.getChild(nodeReg) instanceof MeasurementMNode && idx >=
nodes.length) {
+ cnt++;
+} else {
+ cnt += getDevicesCount(node.getChild(nodeReg), nodes, idx + 1);
+}
+ }
+} else {
+ boolean deviceAdded = false;
+ for (MNode child : node.getChildren().values()) {
+if (child instanceof MeasurementMNode) {
+ if (!deviceAdded) {
+cnt++;
+deviceAdded = true;
+ } else {
+break;
+ }
+} else {
+ cnt += getDevicesCount(child, nodes, idx + 1);
+}
+ }
Review comment:
Ah.. good point. Maybe @yhwang-hbl could add a specific test case for
this example ; )
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] qiaojialin merged pull request #1707: [Hot compaction] Add upgrade logic for TsFileManagement
qiaojialin merged pull request #1707: URL: https://github.com/apache/incubator-iotdb/pull/1707 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] qiaojialin merged pull request #1700: [IOTDB-875]fix flush storage group error when no data in storage group
qiaojialin merged pull request #1700: URL: https://github.com/apache/incubator-iotdb/pull/1700 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] Alima777 commented on a change in pull request #1705: [IOTDB-873] Add count devices DDL
Alima777 commented on a change in pull request #1705:
URL: https://github.com/apache/incubator-iotdb/pull/1705#discussion_r485610150
##
File path:
server/src/test/java/org/apache/iotdb/db/integration/IoTDBMetadataFetchIT.java
##
@@ -118,7 +121,7 @@ public void showTimeseriesTest() throws
ClassNotFoundException, SQLException {
}
Assert.assertEquals(standard, builder.toString());
} catch (SQLException e) {
- e.printStackTrace();
+ logger.error("showTimeseriesTest failed", e);
Review comment:
```suggestion
logger.error("showTimeseriesTest() failed", e);
```
Others same :D
##
File path:
server/src/test/java/org/apache/iotdb/db/integration/IoTDBMetadataFetchIT.java
##
@@ -53,7 +56,7 @@ private static void insertSQL() throws
ClassNotFoundException, SQLException {
statement.execute(sql);
}
} catch (Exception e) {
- e.printStackTrace();
+ logger.error("insertSQL failed", e);
Review comment:
```suggestion
logger.error("insertSQL() failed", e);
```
##
File path: server/src/main/java/org/apache/iotdb/db/metadata/MTree.java
##
@@ -695,6 +708,38 @@ private int getCount(MNode node, String[] nodes, int idx)
throws MetadataExcepti
}
}
+ /**
+ * Traverse the MTree to get the count of devices.
+ */
+ private int getDevicesCount(MNode node, String[] nodes, int idx) throws
MetadataException {
+String nodeReg = MetaUtils.getNodeRegByIdx(idx, nodes);
+int cnt = 0;
+if (!(PATH_WILDCARD).equals(nodeReg)) {
+ if (node.hasChild(nodeReg)) {
+if (node.getChild(nodeReg) instanceof MeasurementMNode && idx >=
nodes.length) {
+ cnt++;
+} else {
+ cnt += getDevicesCount(node.getChild(nodeReg), nodes, idx + 1);
+}
+ }
+} else {
+ boolean deviceAdded = false;
+ for (MNode child : node.getChildren().values()) {
+if (child instanceof MeasurementMNode) {
+ if (!deviceAdded) {
+cnt++;
+deviceAdded = true;
+ } else {
+break;
+ }
+} else {
+ cnt += getDevicesCount(child, nodes, idx + 1);
+}
+ }
Review comment:
Hi, we support subdevice like "root.sg.device1.subdevice1.sensor1" while
subdevice1 is a measurementNode and a device. In your case, the subdevice1 will
be ignored.
You can refer to the Mtree.findDevice() method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] qiaojialin closed pull request #1715: fix the wrong type
qiaojialin closed pull request #1715: URL: https://github.com/apache/incubator-iotdb/pull/1715 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1715: fix the wrong type
sonarcloud[bot] commented on pull request #1715: URL: https://github.com/apache/incubator-iotdb/pull/1715#issuecomment-689554275 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/i ssues?id=apache_incubator-iotdb=1715=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1715=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1715) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1715=new_duplicated_lines_density=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1715=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] Alima777 opened a new pull request #1717: [IOTDB-881] Double quoted meaurement is recognized as constant in Align by device
Alima777 opened a new pull request #1717: URL: https://github.com/apache/incubator-iotdb/pull/1717 This pr fixes: 1. **IOTDB-881 bug:** Double quoted meaurement is recognized as constant in Align by device. Only single quoted measurement should be recognized as constant after double quotes can be used in path. 2. Modify splitPathToDetachedPath() method: support nested double quotes now. 3. Extract some logic in PhysicalGenerator transfrom() method. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] Genius-pig opened a new pull request #1716: basic structure
Genius-pig opened a new pull request #1716: URL: https://github.com/apache/incubator-iotdb/pull/1716 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] Genius-pig opened a new pull request #1715: fix the wrong type
Genius-pig opened a new pull request #1715: URL: https://github.com/apache/incubator-iotdb/pull/1715 fix the wrong type This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #1650: [IOTDB-848] support order by time asc/desc
sonarcloud[bot] commented on pull request #1650: URL: https://github.com/apache/incubator-iotdb/pull/1650#issuecomment-689521389 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/i ssues?id=apache_incubator-iotdb=1650=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [18 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650=new_duplicated_lines_density=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #1650: [IOTDB-848] support order by time asc/desc
sonarcloud[bot] removed a comment on pull request #1650: URL: https://github.com/apache/incubator-iotdb/pull/1650#issuecomment-688209465 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/i ssues?id=apache_incubator-iotdb=1650=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [6 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=1650=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650=new_duplicated_lines_density=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=1650=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] zhanglingzhe0820 commented on a change in pull request #1707: Update upgrade
zhanglingzhe0820 commented on a change in pull request #1707:
URL: https://github.com/apache/incubator-iotdb/pull/1707#discussion_r485549538
##
File path: docs/zh/SystemDesign/StorageEngine/TsFileManagement.md
##
@@ -0,0 +1,223 @@
+
+
+# TsFileManagement -热合并
+
+## 整体思路
+
+现在 iotdb
会因为自动参数优化将很小的数据块写入文件，使得数据文件变得零散化，这导致了用户进行即席查询时需要的读盘次数过多，而后期的合并过程又不是即时进行的，这导致了系统对热数据的查询效率变慢，所以我们借鉴了虚拟内存的思想，在写入流程中增加了热合并过程，通过提高一部分写放大保证了写入到正式文件的数据块不小于给定阈值，提升系统即席查询的效率。将这部分逻辑写到
StorageGroupProcessor 里去，针对封口的 tsfile 文件进行合并。
+
+- iotdb-engine.properties 加一个参数 tsfile_manage_strategy：表示 tsfile 文件管理策略
+- tsfile_manage_strategy 内置两个策略：LevelStrategy 和 NormalStrategy
+- LevelStrategy 即开启热合并并使用层级合并方法，NormalStrategy 即关闭热合并
+- iotdb-engine.properties 中加
merge_chunk_point_number：最大的chunk大小限制，LevelStrategy 时，当达到该参数就合并到最后一层
+- iotdb-engine.properties 加一个参数 max_level_num：LevelStrategy 时最大层数
+- 新建一个 TsFileManagement 类，专门管理 StorageGroupProcessor 中的 seqFileList 和
unSeqFileList ，在这里写热合并的主体逻辑，对外抽象对seqFileList和unSeqFileList的一系列所需接口
+- 对于普通 merge 会调用 TsFileManagement 的 getStableTsFileList()
来获取准备进行文件合并的文件列表，这里对于 LevelStrategy 来说就是返回第 {max_level_num - 1} 层的文件，对于
NormalStrategy 来说就是返回所有文件列表
+- 每一次热合并会取第一个被合并文件的时间戳作为新文件的时间戳，即 {firstFileTimestamp}-{version}-{mergeVerion
+ 1}.tsfiles
+
+## TsFileManagement 对外提供的接口和类
+
+- TsFileManagement 也管理未封口文件
+- TsFileManagement 同时管理 Seq 和 UnSeq 文件列表，在 StorageGroupProcessor 中只创建一个
TsFileManagement
+- List\ getStableTsFileList() 对外提供稳定的 TsFileResource 列表
+- List\ getTsFileList(boolean sequence) 对外提供（顺序/乱序）文件列表（如果
sequence = true，则提供按时间戳顺序的列表）
+- Iterator\ getIterator(boolean sequence) 对外提供（顺序/乱序）文件迭代器（如果
sequence = true，则提供按时间戳顺序的迭代器）
+- void remove(TsFileResource tsFileResource, boolean sequence) 删除对应的
TsFileResource 文件
+- void removeAll(List\ tsfileReourceList, boolean sequence)
删除对应的 TsFileResource 列表
+- void addAll(List\ tsfileReourceList, boolean sequence) 批量加入
TsFileResource 列表
+- boolean isEmpty(boolean sequence) 是否为空
+- int size(boolean sequence) 对应的文件列表长度
+- void forkCurrentFileList(long timePartition) 保存当前文件的某时间分区列表
+- void recover() 调用对应的恢复流程
+- HotCompactionMergeTask 调用对应的异步合并流程，传入 closeUnsealedTsFileProcessorCallBack
以通知外部类合并结束
+
+## LevelStrategy merge 流程
+
+* 外部调用 forkCurrentFileList(long timePartition) 保存当前文件的某时间分区列表
+ * 这里选择的文件列表 chunk 点数之和不超出 merge_chunk_point_number
+* 外部异步创建并提交 merge 线程
+* 判断是否要进行全局热合并
+ * 生成目标文件 {first_file_name}-{max_level_num - 1}.tsfile
+ * 生成合并日志 .hot_compaction.log
+ * 记录是全局合并
+ * 记录目标文件
+ * 进行合并（记录日志 device - offset）
+ * 记录完成合并
+* 循环判断每一层的文件是否要合并到下一层
+ * 生成目标文件 {first_file_name}-{level + 1}.tsfile
+ * 生成合并日志 .hot_compaction.log
+ * 记录是全局合并
Review comment:
I fix it to level
##
File path: docs/zh/SystemDesign/StorageEngine/TsFileManagement.md
##
@@ -0,0 +1,223 @@
+
+
+# TsFileManagement -热合并
+
+## 整体思路
+
+现在 iotdb
会因为自动参数优化将很小的数据块写入文件，使得数据文件变得零散化，这导致了用户进行即席查询时需要的读盘次数过多，而后期的合并过程又不是即时进行的，这导致了系统对热数据的查询效率变慢，所以我们借鉴了虚拟内存的思想，在写入流程中增加了热合并过程，通过提高一部分写放大保证了写入到正式文件的数据块不小于给定阈值，提升系统即席查询的效率。将这部分逻辑写到
StorageGroupProcessor 里去，针对封口的 tsfile 文件进行合并。
+
+- iotdb-engine.properties 加一个参数 tsfile_manage_strategy：表示 tsfile 文件管理策略
+- tsfile_manage_strategy 内置两个策略：LevelStrategy 和 NormalStrategy
+- LevelStrategy 即开启热合并并使用层级合并方法，NormalStrategy 即关闭热合并
+- iotdb-engine.properties 中加
merge_chunk_point_number：最大的chunk大小限制，LevelStrategy 时，当达到该参数就合并到最后一层
+- iotdb-engine.properties 加一个参数 max_level_num：LevelStrategy 时最大层数
+- 新建一个 TsFileManagement 类，专门管理 StorageGroupProcessor 中的 seqFileList 和
unSeqFileList ，在这里写热合并的主体逻辑，对外抽象对seqFileList和unSeqFileList的一系列所需接口
+- 对于普通 merge 会调用 TsFileManagement 的 getStableTsFileList()
来获取准备进行文件合并的文件列表，这里对于 LevelStrategy 来说就是返回第 {max_level_num - 1} 层的文件，对于
NormalStrategy 来说就是返回所有文件列表
+- 每一次热合并会取第一个被合并文件的时间戳作为新文件的时间戳，即 {firstFileTimestamp}-{version}-{mergeVerion
+ 1}.tsfiles
+
+## TsFileManagement 对外提供的接口和类
+
+- TsFileManagement 也管理未封口文件
+- TsFileManagement 同时管理 Seq 和 UnSeq 文件列表，在 StorageGroupProcessor 中只创建一个
TsFileManagement
+- List\ getStableTsFileList() 对外提供稳定的 TsFileResource 列表
+- List\ getTsFileList(boolean sequence) 对外提供（顺序/乱序）文件列表（如果
sequence = true，则提供按时间戳顺序的列表）
+- Iterator\ getIterator(boolean sequence) 对外提供（顺序/乱序）文件迭代器（如果
sequence = true，则提供按时间戳顺序的迭代器）
+- void remove(TsFileResource tsFileResource, boolean sequence) 删除对应的
TsFileResource 文件
+- void removeAll(List\ tsfileReourceList, boolean sequence)
删除对应的 TsFileResource 列表
+- void addAll(List\ tsfileReourceList, boolean sequence) 批量加入
TsFileResource 列表
+- boolean isEmpty(boolean sequence) 是否为空
+- int size(boolean sequence) 对应的文件列表长度
+- void forkCurrentFileList(long timePartition) 保存当前文件的某时间分区列表
+- void recover() 调用对应的恢复流程
+- HotCompactionMergeTask 调用对应的异步合并流程，传入 closeUnsealedTsFileProcessorCallBack
以通知外部类合并结束
+
+## LevelStrategy merge 流程
+
+* 外部调用 forkCurrentFileList(long timePartition) 保存当前文件的某时间分区列表
+ * 这里选择的文件列表 chunk 点数之和不超出 merge_chunk_point_number
+* 外部异步创建并提交 merge 线程
+* 判断是否要进行全局热合并
+ *
[GitHub] [incubator-iotdb] chrisdutz commented on a change in pull request #1714: [IOTDB-869] Failed to insert data via Windows CLI
chrisdutz commented on a change in pull request #1714:
URL: https://github.com/apache/incubator-iotdb/pull/1714#discussion_r485532207
##
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java
##
@@ -118,12 +119,14 @@ public URL getPropsUrl() {
else if(!urlString.endsWith(".properties")) {
urlString += (File.separatorChar + IoTDBConfig.CONFIG_NAME);
}
-// If the url doesn't start with "file:" it's provided as a normal path.
-// So we need to add it to make it a real URL.
-if(!urlString.startsWith("file:")) {
- urlString = "file:" + urlString;
-}
+
+// If the url doesn't start with "file:" or other protocols, it's provided
as a normal path.
+// If valid, we return a URL converted from this file path.
+File confFile = new File(urlString);
try {
+ if (confFile.exists()) {
+return confFile.toURI().toURL();
+ }
Review comment:
This change will again break the ability to load a config properites
resource from classpath ...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #1707: Update upgrade
qiaojialin commented on a change in pull request #1707:
URL: https://github.com/apache/incubator-iotdb/pull/1707#discussion_r485537092
##
File path: docs/zh/SystemDesign/StorageEngine/TsFileManagement.md
##
@@ -0,0 +1,223 @@
+
+
+# TsFileManagement -热合并
+
+## 整体思路
+
+现在 iotdb
会因为自动参数优化将很小的数据块写入文件，使得数据文件变得零散化，这导致了用户进行即席查询时需要的读盘次数过多，而后期的合并过程又不是即时进行的，这导致了系统对热数据的查询效率变慢，所以我们借鉴了虚拟内存的思想，在写入流程中增加了热合并过程，通过提高一部分写放大保证了写入到正式文件的数据块不小于给定阈值，提升系统即席查询的效率。将这部分逻辑写到
StorageGroupProcessor 里去，针对封口的 tsfile 文件进行合并。
+
+- iotdb-engine.properties 加一个参数 tsfile_manage_strategy：表示 tsfile 文件管理策略
+- tsfile_manage_strategy 内置两个策略：LevelStrategy 和 NormalStrategy
+- LevelStrategy 即开启热合并并使用层级合并方法，NormalStrategy 即关闭热合并
+- iotdb-engine.properties 中加
merge_chunk_point_number：最大的chunk大小限制，LevelStrategy 时，当达到该参数就合并到最后一层
+- iotdb-engine.properties 加一个参数 max_level_num：LevelStrategy 时最大层数
+- 新建一个 TsFileManagement 类，专门管理 StorageGroupProcessor 中的 seqFileList 和
unSeqFileList ，在这里写热合并的主体逻辑，对外抽象对seqFileList和unSeqFileList的一系列所需接口
+- 对于普通 merge 会调用 TsFileManagement 的 getStableTsFileList()
来获取准备进行文件合并的文件列表，这里对于 LevelStrategy 来说就是返回第 {max_level_num - 1} 层的文件，对于
NormalStrategy 来说就是返回所有文件列表
+- 每一次热合并会取第一个被合并文件的时间戳作为新文件的时间戳，即 {firstFileTimestamp}-{version}-{mergeVerion
+ 1}.tsfiles
+
+## TsFileManagement 对外提供的接口和类
+
+- TsFileManagement 也管理未封口文件
+- TsFileManagement 同时管理 Seq 和 UnSeq 文件列表，在 StorageGroupProcessor 中只创建一个
TsFileManagement
+- List\ getStableTsFileList() 对外提供稳定的 TsFileResource 列表
+- List\ getTsFileList(boolean sequence) 对外提供（顺序/乱序）文件列表（如果
sequence = true，则提供按时间戳顺序的列表）
+- Iterator\ getIterator(boolean sequence) 对外提供（顺序/乱序）文件迭代器（如果
sequence = true，则提供按时间戳顺序的迭代器）
+- void remove(TsFileResource tsFileResource, boolean sequence) 删除对应的
TsFileResource 文件
+- void removeAll(List\ tsfileReourceList, boolean sequence)
删除对应的 TsFileResource 列表
+- void addAll(List\ tsfileReourceList, boolean sequence) 批量加入
TsFileResource 列表
+- boolean isEmpty(boolean sequence) 是否为空
+- int size(boolean sequence) 对应的文件列表长度
+- void forkCurrentFileList(long timePartition) 保存当前文件的某时间分区列表
+- void recover() 调用对应的恢复流程
+- HotCompactionMergeTask 调用对应的异步合并流程，传入 closeUnsealedTsFileProcessorCallBack
以通知外部类合并结束
+
+## LevelStrategy merge 流程
+
+* 外部调用 forkCurrentFileList(long timePartition) 保存当前文件的某时间分区列表
+ * 这里选择的文件列表 chunk 点数之和不超出 merge_chunk_point_number
+* 外部异步创建并提交 merge 线程
+* 判断是否要进行全局热合并
+ * 生成目标文件 {first_file_name}-{max_level_num - 1}.tsfile
+ * 生成合并日志 .hot_compaction.log
+ * 记录是全局合并
+ * 记录目标文件
+ * 进行合并（记录日志 device - offset）
+ * 记录完成合并
+* 循环判断每一层的文件是否要合并到下一层
+ * 生成目标文件 {first_file_name}-{level + 1}.tsfile
+ * 生成合并日志 .hot_compaction.log
+ * 记录是全局合并
Review comment:
fix
##
File path: docs/zh/SystemDesign/StorageEngine/TsFileManagement.md
##
@@ -0,0 +1,223 @@
+
+
+# TsFileManagement -热合并
+
+## 整体思路
+
+现在 iotdb
会因为自动参数优化将很小的数据块写入文件，使得数据文件变得零散化，这导致了用户进行即席查询时需要的读盘次数过多，而后期的合并过程又不是即时进行的，这导致了系统对热数据的查询效率变慢，所以我们借鉴了虚拟内存的思想，在写入流程中增加了热合并过程，通过提高一部分写放大保证了写入到正式文件的数据块不小于给定阈值，提升系统即席查询的效率。将这部分逻辑写到
StorageGroupProcessor 里去，针对封口的 tsfile 文件进行合并。
+
+- iotdb-engine.properties 加一个参数 tsfile_manage_strategy：表示 tsfile 文件管理策略
+- tsfile_manage_strategy 内置两个策略：LevelStrategy 和 NormalStrategy
+- LevelStrategy 即开启热合并并使用层级合并方法，NormalStrategy 即关闭热合并
+- iotdb-engine.properties 中加
merge_chunk_point_number：最大的chunk大小限制，LevelStrategy 时，当达到该参数就合并到最后一层
+- iotdb-engine.properties 加一个参数 max_level_num：LevelStrategy 时最大层数
+- 新建一个 TsFileManagement 类，专门管理 StorageGroupProcessor 中的 seqFileList 和
unSeqFileList ，在这里写热合并的主体逻辑，对外抽象对seqFileList和unSeqFileList的一系列所需接口
+- 对于普通 merge 会调用 TsFileManagement 的 getStableTsFileList()
来获取准备进行文件合并的文件列表，这里对于 LevelStrategy 来说就是返回第 {max_level_num - 1} 层的文件，对于
NormalStrategy 来说就是返回所有文件列表
+- 每一次热合并会取第一个被合并文件的时间戳作为新文件的时间戳，即 {firstFileTimestamp}-{version}-{mergeVerion
+ 1}.tsfiles
+
+## TsFileManagement 对外提供的接口和类
+
+- TsFileManagement 也管理未封口文件
+- TsFileManagement 同时管理 Seq 和 UnSeq 文件列表，在 StorageGroupProcessor 中只创建一个
TsFileManagement
+- List\ getStableTsFileList() 对外提供稳定的 TsFileResource 列表
+- List\ getTsFileList(boolean sequence) 对外提供（顺序/乱序）文件列表（如果
sequence = true，则提供按时间戳顺序的列表）
+- Iterator\ getIterator(boolean sequence) 对外提供（顺序/乱序）文件迭代器（如果
sequence = true，则提供按时间戳顺序的迭代器）
+- void remove(TsFileResource tsFileResource, boolean sequence) 删除对应的
TsFileResource 文件
+- void removeAll(List\ tsfileReourceList, boolean sequence)
删除对应的 TsFileResource 列表
+- void addAll(List\ tsfileReourceList, boolean sequence) 批量加入
TsFileResource 列表
+- boolean isEmpty(boolean sequence) 是否为空
+- int size(boolean sequence) 对应的文件列表长度
+- void forkCurrentFileList(long timePartition) 保存当前文件的某时间分区列表
+- void recover() 调用对应的恢复流程
+- HotCompactionMergeTask 调用对应的异步合并流程，传入 closeUnsealedTsFileProcessorCallBack
以通知外部类合并结束
+
+## LevelStrategy merge 流程
+
+* 外部调用 forkCurrentFileList(long timePartition) 保存当前文件的某时间分区列表
+ * 这里选择的文件列表 chunk 点数之和不超出 merge_chunk_point_number
+* 外部异步创建并提交 merge 线程
+* 判断是否要进行全局热合并
+ * 生成目标文件
[GitHub] [incubator-iotdb] haimeiguo commented on a change in pull request #1657: Fix for Codesmell check step1
haimeiguo commented on a change in pull request #1657:
URL: https://github.com/apache/incubator-iotdb/pull/1657#discussion_r485482003
##
File path: flink-iotdb-connector/src/main/java/org/apache/iotdb/flink/Event.java
##
@@ -32,8 +32,8 @@
private List types;
private List values;
- public Event(String device, Long timestamp, List measurements,
List types,
- List values) {
+ public Event(String device, final Long timestamp, final List
measurements, final List types,
Review comment:
you are right. we just reverted this commit and deleted 'final' keyword.
Thank you!!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on pull request #460: [IOTDB-68] New shared-nothing cluster
sonarcloud[bot] commented on pull request #460: URL: https://github.com/apache/incubator-iotdb/pull/460#issuecomment-689442314 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/issue s?id=apache_incubator-iotdb=460=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [82 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460=new_duplicated_lines_density=list) [1.1% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on pull request #460: [IOTDB-68] New shared-nothing cluster
sonarcloud[bot] removed a comment on pull request #460: URL: https://github.com/apache/incubator-iotdb/pull/460#issuecomment-689316795 SonarCloud Quality Gate failed. [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) [2 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=VULNERABILITY) (and [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=SECURITY_HOTSPOT) [4 Security Hotspots](https://sonarcloud.io/project/issue s?id=apache_incubator-iotdb=460=false=SECURITY_HOTSPOT) to review) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [153 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb=460=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460=new_duplicated_lines_density=list) [1.1% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb=460=new_duplicated_lines_density=list) The version of Java (1.8.0_252) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11. Read more [here](https://sonarcloud.io/documentation/upcoming/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] sunjincheng121 merged pull request #1711: [IOTDB-878][connector] Bump Flink to Flink 1.11.1
sunjincheng121 merged pull request #1711: URL: https://github.com/apache/incubator-iotdb/pull/1711 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] wshao08 opened a new pull request #1714: [IOTDB-869] Failed to insert data via Windows CLI
wshao08 opened a new pull request #1714: URL: https://github.com/apache/incubator-iotdb/pull/1714 This issue occurs because IoTDB failed to load iotdb-engine.properties file on Windows. When the file path is in the format of "D:/path" on WinOS, it will not be recognized as valid protocol when creating a URL object. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-iotdb] SimonSimCity opened a new pull request #1713: Python method "set_time_zone" exception message
SimonSimCity opened a new pull request #1713: URL: https://github.com/apache/incubator-iotdb/pull/1713 The method `set_time_zone` of the :snake: client threw an exception with a message mentioning `get time zone`. I guess that's a typo, so I fixed it :sunglasses: Are there tests for this client? I couldn't find any :worried: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
