[jira] [Created] (IOTDB-1131) dictionary encoding of deviceID and measurementID in WAL
Xiangdong Huang created IOTDB-1131: -- Summary: dictionary encoding of deviceID and measurementID in WAL Key: IOTDB-1131 URL: https://issues.apache.org/jira/browse/IOTDB-1131 Project: Apache IoTDB Issue Type: Improvement Components: WAL Reporter: Xiangdong Huang This is an interesting idea that proposed by Tian Jiang. Copy from Tian Jiang: Write ahead logs (WALs) ensure that data which are not persisted yet can still be recovered from a system failure, thus to increase the durability of a DBMS. However, WALs generally require more frequent flushes to limit the possibility of losing data, which increases disk utilities significantly as each flush requires one disk I/O. Moreover, logs are hardly compressed or encoded like what we are doing to the raw data in TsFiles, and result is that logs containing the same data consume much larger space than the data chunks. The disadvantages are two-folds: first, large logs will compete for more disk bandwidth, slowing down the persistence of raw data; second, even if WALs are placed on another disk, (possibly SSD for high throughput), as WALs are removed frequently once their corresponding data are persisted, such frequent write-and-erases will shorten disk life especially for SSDs. So it is beneficial to reduce the sizes of WALs. In IoTDB (and also other DBMSs), the majority of WALs are logs of insertions, as other operations like deletions and updates are often rare compared with insertions. This observation enlightens us that may focus on reducing sizes of insertion logs, which is enough to attain ideal improvement of the whole system. Currently, we serialize complete physical plans into WAL, but we notice that despite values and timestamps generally varies from plan to plan, head information like deviceIds, measurementIds and data types are highly redundant, and sometimes deviceIds and measurementIds are long strings, which may consume a significant amount of space. So in this design, we concentrate on reducing duplicated deviceIds, measurementIds and data types in WALs. Method To reduce duplicated deviceIds, measurementIds and data types in WALs, we use windowed differentiation technique (or referencing) to replace redundant fields with a index pointing to a base log, if such a log can be found within a given window. Detailed procedure are described below: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1131) dictionary encoding of deviceID and measurementID in WAL
[ https://issues.apache.org/jira/browse/IOTDB-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272686#comment-17272686 ] Xiangdong Huang commented on IOTDB-1131: see https://github.com/apache/iotdb/pull/1869 for the draft. > dictionary encoding of deviceID and measurementID in WAL > > > Key: IOTDB-1131 > URL: https://issues.apache.org/jira/browse/IOTDB-1131 > Project: Apache IoTDB > Issue Type: Improvement > Components: WAL >Reporter: Xiangdong Huang >Priority: Major > > This is an interesting idea that proposed by Tian Jiang. > Copy from Tian Jiang: > Write ahead logs (WALs) ensure that data which are not persisted yet can > still be recovered from a system failure, thus to increase the durability of > a DBMS. However, WALs generally require more frequent flushes to limit the > possibility of losing data, which increases disk utilities significantly as > each flush requires one disk I/O. Moreover, logs are hardly compressed or > encoded like what we are doing to the raw data in TsFiles, and result is that > logs containing the same data consume much larger space than the data chunks. > The disadvantages are two-folds: first, large logs will compete for more disk > bandwidth, slowing down the persistence of raw data; second, even if WALs are > placed on another disk, (possibly SSD for high throughput), as WALs are > removed frequently once their corresponding data are persisted, such frequent > write-and-erases will shorten disk life especially for SSDs. > So it is beneficial to reduce the sizes of WALs. In IoTDB (and also other > DBMSs), the majority of WALs are logs of insertions, as other operations like > deletions and updates are often rare compared with insertions. This > observation enlightens us that may focus on reducing sizes of insertion logs, > which is enough to attain ideal improvement of the whole system. Currently, > we serialize complete physical plans into WAL, but we notice that despite > values and timestamps generally varies from plan to plan, head information > like deviceIds, measurementIds and data types are highly redundant, and > sometimes deviceIds and measurementIds are long strings, which may consume a > significant amount of space. So in this design, we concentrate on reducing > duplicated deviceIds, measurementIds and data types in WALs. > Method > To reduce duplicated deviceIds, measurementIds and data types in WALs, we use > windowed differentiation technique (or referencing) to replace redundant > fields with a index pointing to a base log, if such a log can be found within > a given window. Detailed procedure are described below: -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-1132) The password of create user statement contains a blank space can be created successfully and the user can login in database
刘珍 created IOTDB-1132: - Summary: The password of create user statement contains a blank space can be created successfully and the user can login in database Key: IOTDB-1132 URL: https://issues.apache.org/jira/browse/IOTDB-1132 Project: Apache IoTDB Issue Type: Bug Components: Server Environment: 0.11.3-SNAPSHOT Reporter: 刘珍 CREATE USER u_test 'write_ pwd' Msg: The statement is executed successfully. 密码中包含1个空格 用户u_test登陆数据库，输入密码（包含空格），成功。 用户手册： h3. 密码限制 IoTDB规定密码的字符长度不小于4，其中密码不能包含空格，密码采用MD5进行加密。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-1133) Reduce the code compilation time
Houliang Qi created IOTDB-1133: -- Summary: Reduce the code compilation time Key: IOTDB-1133 URL: https://issues.apache.org/jira/browse/IOTDB-1133 Project: Apache IoTDB Issue Type: Improvement Components: Others Reporter: Houliang Qi Fix For: 0.12.0 Attachments: image-2021-01-27-19-31-44-993.png, image-2021-01-27-19-31-51-791.png Now, due to the introduction of vulnerabilities check, the code compilation time is greatly increased, which makes developers need to wait a lot of time for the compilation to complete when testing. I think it's better to add one configuration when building the codes , so that when testing, we can disable the vulnerabilities check. And we can see from the result that, when disable the vulnerabilities check, the compilation speed is more than doubled. !image-2021-01-27-19-28-23-110.png! !image-2021-01-27-19-28-40-603.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IOTDB-1133) Reduce the code compilation time
[ https://issues.apache.org/jira/browse/IOTDB-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Houliang Qi reassigned IOTDB-1133: -- Assignee: Houliang Qi > Reduce the code compilation time > > > Key: IOTDB-1133 > URL: https://issues.apache.org/jira/browse/IOTDB-1133 > Project: Apache IoTDB > Issue Type: Improvement > Components: Others >Reporter: Houliang Qi >Assignee: Houliang Qi >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > Attachments: image-2021-01-27-19-31-44-993.png, > image-2021-01-27-19-31-51-791.png > > > Now, due to the introduction of vulnerabilities check, the code compilation > time is greatly increased, which makes developers need to wait a lot of time > for the compilation to complete when testing. I think it's better to add one > configuration when building the codes , so that when testing, we can disable > the vulnerabilities check. > And we can see from the result that, when disable the vulnerabilities check, > the compilation speed is more than doubled. > > !image-2021-01-27-19-31-44-993.png! > !image-2021-01-27-19-31-51-791.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1126) unseq tsfile delete due to merge
[ https://issues.apache.org/jira/browse/IOTDB-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272912#comment-17272912 ] Chao Wang commented on IOTDB-1126: -- Thanks for reivew and merged, will 0.12.0 release > unseq tsfile delete due to merge > > > Key: IOTDB-1126 > URL: https://issues.apache.org/jira/browse/IOTDB-1126 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > I find , if unseq tsfile and no seq-tsfile merge, so unseq tsfile delete and > do not generate merge file . as follow : > set storage group to root.test > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > DELETE PARTITION root.test 0; > select * from root.test.wf02.wt02; > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > flush > select * from root.test.wf02.wt02; > this is null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1126) unseq tsfile delete due to merge
[ https://issues.apache.org/jira/browse/IOTDB-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272920#comment-17272920 ] Xiangdong Huang commented on IOTDB-1126: Hi [~wangchao316], This PR is good. but one more question: If all data in seq folder is deleted, why not consider new arrival data as sequence data? > unseq tsfile delete due to merge > > > Key: IOTDB-1126 > URL: https://issues.apache.org/jira/browse/IOTDB-1126 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > I find , if unseq tsfile and no seq-tsfile merge, so unseq tsfile delete and > do not generate merge file . as follow : > set storage group to root.test > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > DELETE PARTITION root.test 0; > select * from root.test.wf02.wt02; > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > flush > select * from root.test.wf02.wt02; > this is null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1126) unseq tsfile delete due to merge
[ https://issues.apache.org/jira/browse/IOTDB-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272931#comment-17272931 ] Chao Wang commented on IOTDB-1126: -- [~hxd] hello , Thanks . in this PR, all data in seq folder is deleted, will update lasterflushtime to Long.MIN_VALUE, so new insert data will be sequence data. your idea is solved in this PR. > unseq tsfile delete due to merge > > > Key: IOTDB-1126 > URL: https://issues.apache.org/jira/browse/IOTDB-1126 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > I find , if unseq tsfile and no seq-tsfile merge, so unseq tsfile delete and > do not generate merge file . as follow : > set storage group to root.test > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > DELETE PARTITION root.test 0; > select * from root.test.wf02.wt02; > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > flush > select * from root.test.wf02.wt02; > this is null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1126) unseq tsfile delete due to merge
[ https://issues.apache.org/jira/browse/IOTDB-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272932#comment-17272932 ] Xiangdong Huang commented on IOTDB-1126: OK. > unseq tsfile delete due to merge > > > Key: IOTDB-1126 > URL: https://issues.apache.org/jira/browse/IOTDB-1126 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > I find , if unseq tsfile and no seq-tsfile merge, so unseq tsfile delete and > do not generate merge file . as follow : > set storage group to root.test > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > DELETE PARTITION root.test 0; > select * from root.test.wf02.wt02; > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > flush > select * from root.test.wf02.wt02; > this is null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1126) unseq tsfile delete due to merge
[ https://issues.apache.org/jira/browse/IOTDB-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273252#comment-17273252 ] Chao Wang commented on IOTDB-1126: -- cherry pick, merged to 0.11 branch , will make 0.11.3 release > unseq tsfile delete due to merge > > > Key: IOTDB-1126 > URL: https://issues.apache.org/jira/browse/IOTDB-1126 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > I find , if unseq tsfile and no seq-tsfile merge, so unseq tsfile delete and > do not generate merge file . as follow : > set storage group to root.test > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > DELETE PARTITION root.test 0; > select * from root.test.wf02.wt02; > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > flush > select * from root.test.wf02.wt02; > this is null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-1134) cluster - show devices not consistent
Haimei Guo created IOTDB-1134:
-
Summary: cluster - show devices not consistent
Key: IOTDB-1134
URL: https://issues.apache.org/jira/browse/IOTDB-1134
Project: Apache IoTDB
Issue Type: Bug
Reporter: Haimei Guo
I created 20 devices in total. When connect to different node, the SHOW DEVICES
result is not consistent.
{code:java}
// code placeholder
{code}
IoTDB> show devices +--+ | devices| +--+ |
root.group_1.d_1| |root.group_10.d_10| |root.group_12.d_12|
|root.group_13.d_13| |root.group_15.d_15| |root.group_16.d_16|
|root.group_18.d_18| |root.group_19.d_19| | root.group_3.d_3| |
root.group_4.d_4| | root.group_7.d_7| | root.group_9.d_9| +--+
Total line number = 12
{code:java}
// code placeholder
{code}
IoTDB> show devices +--+ | devices| +--+ |
root.group_0.d_0| | root.group_1.d_1| |root.group_10.d_10| |root.group_11.d_11|
|root.group_12.d_12| |root.group_13.d_13| |root.group_14.d_14|
|root.group_16.d_16| |root.group_17.d_17| |root.group_18.d_18|
|root.group_19.d_19| | root.group_2.d_2| | root.group_3.d_3| |
root.group_4.d_4| | root.group_5.d_5| | root.group_6.d_6| | root.group_7.d_7| |
root.group_8.d_8| | root.group_9.d_9| +--+ Total line number =
19
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (IOTDB-1085) Refactoring "transformQuery" method
[ https://issues.apache.org/jira/browse/IOTDB-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiangweiWei reassigned IOTDB-1085: -- Assignee: XiangweiWei > Refactoring "transformQuery" method > --- > > Key: IOTDB-1085 > URL: https://issues.apache.org/jira/browse/IOTDB-1085 > Project: Apache IoTDB > Issue Type: Improvement > Components: Others >Reporter: weizihan >Assignee: XiangweiWei >Priority: Critical > Attachments: image-2020-12-28-14-11-22-516.png, > image-2020-12-28-14-21-36-561.png, image-2020-12-28-14-22-04-775.png, > image-2020-12-28-14-22-35-650.png > > > Currently, the method of > org.apache.iotdb.db.qp.strategy.PhysicalGenerator#transformQuery has reached > about 300 line, and there are lots of "if else" in this method. In order to > easily maintain it, it should be refactored into more readable one, I think. > > Related pic: > !image-2020-12-28-14-11-22-516.png|width=518,height=60! > !image-2020-12-28-14-21-36-561.png|width=553,height=710! > !image-2020-12-28-14-22-04-775.png|width=539,height=670! > > !image-2020-12-28-14-22-35-650.png|width=571,height=761! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1085) Refactoring "transformQuery" method
[ https://issues.apache.org/jira/browse/IOTDB-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273261#comment-17273261 ] XiangweiWei commented on IOTDB-1085: It has been in progress now actually. Thanks for your suggestion. > Refactoring "transformQuery" method > --- > > Key: IOTDB-1085 > URL: https://issues.apache.org/jira/browse/IOTDB-1085 > Project: Apache IoTDB > Issue Type: Improvement > Components: Others >Reporter: weizihan >Assignee: XiangweiWei >Priority: Critical > Attachments: image-2020-12-28-14-11-22-516.png, > image-2020-12-28-14-21-36-561.png, image-2020-12-28-14-22-04-775.png, > image-2020-12-28-14-22-35-650.png > > > Currently, the method of > org.apache.iotdb.db.qp.strategy.PhysicalGenerator#transformQuery has reached > about 300 line, and there are lots of "if else" in this method. In order to > easily maintain it, it should be refactored into more readable one, I think. > > Related pic: > !image-2020-12-28-14-11-22-516.png|width=518,height=60! > !image-2020-12-28-14-21-36-561.png|width=553,height=710! > !image-2020-12-28-14-22-04-775.png|width=539,height=670! > > !image-2020-12-28-14-22-35-650.png|width=571,height=761! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1126) unseq tsfile delete due to merge
[ https://issues.apache.org/jira/browse/IOTDB-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273271#comment-17273271 ] Xiangdong Huang commented on IOTDB-1126: Hi, if it needs to cherry-pick to 0.11 branch, pls modify the issue description: - add 0.11.2 to Affects Version/s - once merged, add 0.11.3 to Fix Version/s: > unseq tsfile delete due to merge > > > Key: IOTDB-1126 > URL: https://issues.apache.org/jira/browse/IOTDB-1126 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.12.0 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > I find , if unseq tsfile and no seq-tsfile merge, so unseq tsfile delete and > do not generate merge file . as follow : > set storage group to root.test > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > DELETE PARTITION root.test 0; > select * from root.test.wf02.wt02; > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > flush > select * from root.test.wf02.wt02; > this is null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IOTDB-1126) unseq tsfile delete due to merge
[ https://issues.apache.org/jira/browse/IOTDB-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273277#comment-17273277 ] Chao Wang commented on IOTDB-1126: -- good， Thanks ... > unseq tsfile delete due to merge > > > Key: IOTDB-1126 > URL: https://issues.apache.org/jira/browse/IOTDB-1126 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.12.0, 0.11.2 >Reporter: Chao Wang >Assignee: Chao Wang >Priority: Major > Labels: pull-request-available > Fix For: 0.12.0 > > > I find , if unseq tsfile and no seq-tsfile merge, so unseq tsfile delete and > do not generate merge file . as follow : > set storage group to root.test > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > DELETE PARTITION root.test 0; > select * from root.test.wf02.wt02; > insert into root.test.wf02.wt02(timestamp,status) values(1,true) > select * from root.test.wf02.wt02; > flush > select * from root.test.wf02.wt02; > this is null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-1135) count timeseries prefix path bug
Haimei Guo created IOTDB-1135: - Summary: count timeseries prefix path bug Key: IOTDB-1135 URL: https://issues.apache.org/jira/browse/IOTDB-1135 Project: Apache IoTDB Issue Type: Bug Reporter: Haimei Guo when create with same , then execute count timeseries . the result is incorrect. IoTDB> show timeseries root.ln.wf.wt.tdouble +--+-+-+++---++--+ | timeseries|alias|storage group|dataType|encoding|compression|tags|attributes| +--+-+-+++---++--+ | root.ln.wf.wt.tdouble| null| root.ln| INT32| RLE| SNAPPY|null| null| |root.ln.wf.wt.tdouble.tdouble1| null| root.ln| INT32| RLE| SNAPPY|null| null| | root.ln.wf.wt.tdouble.tdouble| null| root.ln| INT32| RLE| SNAPPY|null| null| +--+-+-+++---++--+ Total line number = 3 IoTDB> count timeseries root.ln.wf.wt.tdouble +-+ |count| +-+ | 1| +-+ Total line number = 1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
