[jira] [Created] (IOTDB-3168) The path with * could be executed successfully when inserting
Wei Liu created IOTDB-3168:
--
Summary: The path with * could be executed successfully when
inserting
Key: IOTDB-3168
URL: https://issues.apache.org/jira/browse/IOTDB-3168
Project: Apache IoTDB
Issue Type: Bug
Reporter: Wei Liu
Fix For: 0.14.0, 0.13.1
IoTDB> create timeseries root.test.*.a.b
Msg: 401: Error occurred while parsing SQL to physical plan: line 1:28
mismatched input '*' expecting \{ID, QUOTED_ID}
IoTDB> insert into root.test.*.a(timestamp, b) values(150946560, true)
Msg: The statement is executed successfully.
IoTDB> show timeseries root.test.**
+---+-+-+++---++--+
| timeseries|alias|storage
group|dataType|encoding|compression|tags|attributes|
+---+-+-+++---++--+
|root.test.*.a.b| null| root.test| BOOLEAN| RLE| SNAPPY|null|
null|
+---+-+-+++---++--+
Total line number = 1
It costs 0.033s
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (IOTDB-3045) [ schema recovery ] The query result contains some timeseries that have been deleted
[
https://issues.apache.org/jira/browse/IOTDB-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haiming Zhu reassigned IOTDB-3045:
--
Assignee: Haiming Zhu (was: ZhaoXin)
> [ schema recovery ] The query result contains some timeseries that have been
> deleted
>
>
> Key: IOTDB-3045
> URL: https://issues.apache.org/jira/browse/IOTDB-3045
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Haiming Zhu
>Priority: Major
> Attachments: config.properties, d1.log
>
>
> master_0428_ca4f3cf
> 问题描述：
> 50 sg，100 dev，20 sensors/dev, 共2000万序列。
> 100个设备名称记录在文件中，
> 遍历文件，10个用户并发 delete ts，1个设备上delete 1条序列。
> 重复10次，也就是这100个dev，每个dev共delete 10个ts。
> delete 后，执行count timeseries dev.* ；结果符合预期。
> 备份数据。
> {color:#DE350B}重启iotdb，恢复元数据后，再查询 100个dev的序列数，有14个设备的序列数都是20万，
> 期望是10，delete的ts，又查询到了。{color}
> 详细测试流程：
> 1. 私有云 172.20.70.2(iotdb)/16(bm) 8C32G
> iotdb参数
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> schema_engine_mode=Schema_File
> 2. benchmark 配置见附件
> 3. 获取dev 名称，记录到文件dev_name.txt
> cat get_dev_name.sh
> #!/bin/bash
> ./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
> dev_name.txt
> 4. 并发delete
> cat del_1_ts.sh
> #!/bin/bash
> function delete_ts()
> {
>del_ts=$1
>dev_name=$2
>exp_count=$3
>count_ts ${exp_count} ${dev_name}
>./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
>let exp_count--
>count_ts ${exp_count} ${dev_name}
> }
> function count_ts()
> {
> exp_c_ts=$1
> dev_name=$2
> cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
> "4,4p"|awk -F '|' '{print $2}'`
> if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
> echo "ok"
> else
> echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
> fi
> }
> function execute()
> {
> in_file=$1
> i=$2
> exp_count=$3
> n=0
> cat ${in_file} | while read line
> do
>del_ts="${line}.s_${i}"
>dev_name="${line}.*";
>delete_ts ${del_ts} ${dev_name} ${exp_count} &
>let i++
>let n++
>b=$(( $n % 10 ))
>if [ "$b" -eq "0" ];then
> wait
>fi
> done
> wait
> }
> # 循环几轮
> count_ts_base=20
> for loop in {1..10}
> do
> execute "dev_name.txt" ${loop} ${count_ts_base}
> let count_ts_base--
> done
> 并发delete脚本的日志文件见附件。
> 执行count_ts.sh，查看每个dev的count timeseries，是否都是10。
> 有14个dev的结果是20，不符合预期结果。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (IOTDB-3045) [ schema recovery ] The query result contains some timeseries that have been deleted
[
https://issues.apache.org/jira/browse/IOTDB-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535802#comment-17535802
]
刘珍 commented on IOTDB-3045:
---
重启前执行flush，表现是正确的（delete的 查询不到）。
> [ schema recovery ] The query result contains some timeseries that have been
> deleted
>
>
> Key: IOTDB-3045
> URL: https://issues.apache.org/jira/browse/IOTDB-3045
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: ZhaoXin
>Priority: Major
> Attachments: config.properties, d1.log
>
>
> master_0428_ca4f3cf
> 问题描述：
> 50 sg，100 dev，20 sensors/dev, 共2000万序列。
> 100个设备名称记录在文件中，
> 遍历文件，10个用户并发 delete ts，1个设备上delete 1条序列。
> 重复10次，也就是这100个dev，每个dev共delete 10个ts。
> delete 后，执行count timeseries dev.* ；结果符合预期。
> 备份数据。
> {color:#DE350B}重启iotdb，恢复元数据后，再查询 100个dev的序列数，有14个设备的序列数都是20万，
> 期望是10，delete的ts，又查询到了。{color}
> 详细测试流程：
> 1. 私有云 172.20.70.2(iotdb)/16(bm) 8C32G
> iotdb参数
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> schema_engine_mode=Schema_File
> 2. benchmark 配置见附件
> 3. 获取dev 名称，记录到文件dev_name.txt
> cat get_dev_name.sh
> #!/bin/bash
> ./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
> dev_name.txt
> 4. 并发delete
> cat del_1_ts.sh
> #!/bin/bash
> function delete_ts()
> {
>del_ts=$1
>dev_name=$2
>exp_count=$3
>count_ts ${exp_count} ${dev_name}
>./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
>let exp_count--
>count_ts ${exp_count} ${dev_name}
> }
> function count_ts()
> {
> exp_c_ts=$1
> dev_name=$2
> cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
> "4,4p"|awk -F '|' '{print $2}'`
> if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
> echo "ok"
> else
> echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
> fi
> }
> function execute()
> {
> in_file=$1
> i=$2
> exp_count=$3
> n=0
> cat ${in_file} | while read line
> do
>del_ts="${line}.s_${i}"
>dev_name="${line}.*";
>delete_ts ${del_ts} ${dev_name} ${exp_count} &
>let i++
>let n++
>b=$(( $n % 10 ))
>if [ "$b" -eq "0" ];then
> wait
>fi
> done
> wait
> }
> # 循环几轮
> count_ts_base=20
> for loop in {1..10}
> do
> execute "dev_name.txt" ${loop} ${count_ts_base}
> let count_ts_base--
> done
> 并发delete脚本的日志文件见附件。
> 执行count_ts.sh，查看每个dev的count timeseries，是否都是10。
> 有14个dev的结果是20，不符合预期结果。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (IOTDB-3160) TsFile will be corrupted when flushing memtable appears OOM.
[ https://issues.apache.org/jira/browse/IOTDB-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jialin Qiao reassigned IOTDB-3160: -- Assignee: Haiming Zhu > TsFile will be corrupted when flushing memtable appears OOM. > - > > Key: IOTDB-3160 > URL: https://issues.apache.org/jira/browse/IOTDB-3160 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: master branch, 0.13.0 >Reporter: 周沛辰 >Assignee: Haiming Zhu >Priority: Major > Attachments: image-2022-05-11-16-59-49-557.png, > image-2022-05-11-17-08-56-755.png > > > When flushing memtable appears OOM, the file will be corrupted with metadata > only and cannot be repaired at the next restart. The result may be as > following: > When an exception is caught in the flushing memtable, the following logic > will be executed to call the TsFileIOWriter's endfile, causing the file to > have only the metadata area but no data area. After the next restart, > memTable is considered to have flushed properly and WAL is not used to > restore tsfile. > !image-2022-05-11-16-59-49-557.png! > !image-2022-05-11-17-08-56-755.png|width=1594,height=592! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3167) Add wait_leader_ready logic for RatisConsensus
Xinyu Tan created IOTDB-3167: Summary: Add wait_leader_ready logic for RatisConsensus Key: IOTDB-3167 URL: https://issues.apache.org/jira/browse/IOTDB-3167 Project: Apache IoTDB Issue Type: Improvement Reporter: Xinyu Tan Assignee: Xinyu Tan In the Ratis implementation, the new leader needs to commit a log with the latest configuration to commit the operations of the previous term. Before the log is applied, even though the node is the leader, the write may still fail, so we need to add blocking wait logic. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3166) Internal Client Pool lifecycle issue when node restarts
Jinrui Zhang created IOTDB-3166: --- Summary: Internal Client Pool lifecycle issue when node restarts Key: IOTDB-3166 URL: https://issues.apache.org/jira/browse/IOTDB-3166 Project: Apache IoTDB Issue Type: Improvement Reporter: Jinrui Zhang Currently we cached TClient for each EndPoint. But if the corresponding Node restarts, the cached TClient would be invalid due to broken pipe. Next time this client is invoked, the RPC operation will fail. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3165) Implement data deletion in distributed system
周沛辰 created IOTDB-3165: -- Summary: Implement data deletion in distributed system Key: IOTDB-3165 URL: https://issues.apache.org/jira/browse/IOTDB-3165 Project: Apache IoTDB Issue Type: New Feature Reporter: 周沛辰 Assignee: 周沛辰 Implement data deletion in distributed system. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3164) Optimize memory allocation of the system and memory control of compaction
周沛辰 created IOTDB-3164: -- Summary: Optimize memory allocation of the system and memory control of compaction Key: IOTDB-3164 URL: https://issues.apache.org/jira/browse/IOTDB-3164 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3163) Refactor compaction memory estimator frame and add new estimator of RewriteCrossCompaction
周沛辰 created IOTDB-3163: -- Summary: Refactor compaction memory estimator frame and add new estimator of RewriteCrossCompaction Key: IOTDB-3163 URL: https://issues.apache.org/jira/browse/IOTDB-3163 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3162) Split large target files in cross space compaction
周沛辰 created IOTDB-3162: -- Summary: Split large target files in cross space compaction Key: IOTDB-3162 URL: https://issues.apache.org/jira/browse/IOTDB-3162 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 Split large target files in cross space compaction with new rule of tsfile version. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3161) Add under the of iotdb official website.
LiKai created IOTDB-3161: Summary: Add under the of iotdb official website. Key: IOTDB-3161 URL: https://issues.apache.org/jira/browse/IOTDB-3161 Project: Apache IoTDB Issue Type: Improvement Components: Document Reporter: LiKai Assignee: LiKai Attachments: image-2022-05-11-17-14-29-789.png !image-2022-05-11-17-14-29-789.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3160) TsFile will be corrupted when flushing memtable appears OOM.
周沛辰 created IOTDB-3160: -- Summary: TsFile will be corrupted when flushing memtable appears OOM. Key: IOTDB-3160 URL: https://issues.apache.org/jira/browse/IOTDB-3160 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.13.0, master branch Reporter: 周沛辰 Attachments: image-2022-05-11-16-59-49-557.png, image-2022-05-11-17-08-56-755.png When flushing memtable appears OOM, the file will be corrupted with metadata only and cannot be repaired at the next restart. The result may be as following: When an exception is caught in the flushing memtable, the following logic will be executed to call the TsFileIOWriter's endfile, causing the file to have only the metadata area but no data area. After the next restart, memTable is considered to have flushed properly and WAL is not used to restore tsfile. !image-2022-05-11-16-59-49-557.png! !image-2022-05-11-17-08-56-755.png|width=1594,height=592! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3159) Granting the user privileges to create_user does not require adding a path
xiaozhihong created IOTDB-3159:
--
Summary: Granting the user privileges to create_user does not
require adding a path
Key: IOTDB-3159
URL: https://issues.apache.org/jira/browse/IOTDB-3159
Project: Apache IoTDB
Issue Type: Improvement
Components: Core/Server
Reporter: xiaozhihong
Assignee: Jialin Qiao
{code:java}
GRANT USER user01 PRIVILEGES CREATE_USER on root; {code}
For users, the operation privileges related to users and roles have nothing to
do with the path, so there is no need to add on root.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (IOTDB-3136) When the weekly test performs mixed read and write operations, the query reports 500
[ https://issues.apache.org/jira/browse/IOTDB-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534769#comment-17534769 ] Minghui Liu commented on IOTDB-3136: There are some incompatible statements from v0.12 to v0.13 and later. The *IoTDB-013-SESSION* configuration is required when using the benchmark to test v0.13 and master. > When the weekly test performs mixed read and write operations, the query > reports 500 > > > Key: IOTDB-3136 > URL: https://issues.apache.org/jira/browse/IOTDB-3136 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Server >Affects Versions: master branch >Reporter: xiaozhihong >Assignee: Minghui Liu >Priority: Major > Attachments: image-2022-05-10-10-51-35-047.png, > rw_no_overflow.config.properties > > > !image-2022-05-10-10-51-35-047.png! > step 1: start server > step2: start benchmark > step3: check log > For benchmark conf file, please refer to the attachment. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3158) [Metric] The problem when use iotdb-reporter
张洪胤 created IOTDB-3158: -- Summary: [Metric] The problem when use iotdb-reporter Key: IOTDB-3158 URL: https://issues.apache.org/jira/browse/IOTDB-3158 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 When use iotdb-reporter, metric module will stop to write metric into iotdb after a period of time. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3157) [Benchmark] Fix the problem of schema module of benchmark
张洪胤 created IOTDB-3157: -- Summary: [Benchmark] Fix the problem of schema module of benchmark Key: IOTDB-3157 URL: https://issues.apache.org/jira/browse/IOTDB-3157 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 Now iotdb-benchmark use multi-thread to register schema, so we need to control all threads to avoid the repeat register of schema. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3156) [ compaction ] Uneven distribution of merge tasks per sg
刘珍 created IOTDB-3156:
-
Summary: [ compaction ] Uneven distribution of merge tasks per sg
Key: IOTDB-3156
URL: https://issues.apache.org/jira/browse/IOTDB-3156
Project: Apache IoTDB
Issue Type: Improvement
Components: Core/Compaction
Reporter: 刘珍
Assignee: Liuxuxin
Attachments: image-2022-05-11-15-45-33-156.png,
image-2022-05-11-15-45-38-071.png
问题描述 ：
0.13.0 rc1 每个sg的合并任务分布不均匀。
!image-2022-05-11-15-45-33-156.png!
!image-2022-05-11-15-45-38-071.png!
复现流程：
长测配置，顺序数据。
启动测试后，运行脚本：
cat get_sg_tsfile-count.sh
#!/bin/bash
sg_res=" "
function sg_tsfile()
{
tsfile_count=`ls -l
/data/iotdb_data/data/sequence/root.test.g_${1}/0/0/*.resource|wc -l`
sg_res="${sg_res} ${tsfile_count}"
}
while true
do
for i in {0..49}
do
sg_tsfile ${i}
done
time=`date "+%Y-%m-%d %H:%M:%S"`
echo "${time} ${sg_res} " >> sg_tsfile_res_0509.out
sleep 2h
sg_res=" "
done
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
