[jira] [Created] (IOTDB-3015) Authenticate login via confignode
任宇华 created IOTDB-3015: -- Summary: Authenticate login via confignode Key: IOTDB-3015 URL: https://issues.apache.org/jira/browse/IOTDB-3015 Project: Apache IoTDB Issue Type: New Feature Components: Core/Authority Reporter: 任宇华 Fix For: master branch -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-3015) Authenticate login via confignode
[ https://issues.apache.org/jira/browse/IOTDB-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 任宇华 reassigned IOTDB-3015: -- Assignee: 任宇华 > Authenticate login via confignode > - > > Key: IOTDB-3015 > URL: https://issues.apache.org/jira/browse/IOTDB-3015 > Project: Apache IoTDB > Issue Type: New Feature > Components: Core/Authority >Reporter: 任宇华 >Assignee: 任宇华 >Priority: Major > Fix For: master branch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3019) [MPP] Implementation of show storage group.
张洪胤 created IOTDB-3019: -- Summary: [MPP] Implementation of show storage group. Key: IOTDB-3019 URL: https://issues.apache.org/jira/browse/IOTDB-3019 Project: Apache IoTDB Issue Type: New Feature Reporter: 张洪胤 Assignee: 张洪胤 implement: show storage group. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3020) Create schema is slow
刘珍 created IOTDB-3020: - Summary: Create schema is slow Key: IOTDB-3020 URL: https://issues.apache.org/jira/browse/IOTDB-3020 Project: Apache IoTDB Issue Type: Bug Components: Core/Schema Manager Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Jialin Qiao Attachments: image-2022-04-27-15-24-15-249.png master_0427_4bf3dbb 问题描述： sensor_number 大，create schema会非常慢： !image-2022-04-27-15-24-15-249.png! 8C32G/schema_engine_mode=Schema_File fit机器192.168.130.1/2 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3025) add permission check
任宇华 created IOTDB-3025: -- Summary: add permission check Key: IOTDB-3025 URL: https://issues.apache.org/jira/browse/IOTDB-3025 Project: Apache IoTDB Issue Type: New Feature Components: Core/Authority Reporter: 任宇华 Fix For: master branch -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-3025) add permission check
[ https://issues.apache.org/jira/browse/IOTDB-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 任宇华 reassigned IOTDB-3025: -- Assignee: 任宇华 > add permission check > > > Key: IOTDB-3025 > URL: https://issues.apache.org/jira/browse/IOTDB-3025 > Project: Apache IoTDB > Issue Type: New Feature > Components: Core/Authority >Reporter: 任宇华 >Assignee: 任宇华 >Priority: Major > Fix For: master branch > > > add permission check -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3027) [MPP] Implementation of count storage group.
张洪胤 created IOTDB-3027: -- Summary: [MPP] Implementation of count storage group. Key: IOTDB-3027 URL: https://issues.apache.org/jira/browse/IOTDB-3027 Project: Apache IoTDB Issue Type: New Feature Reporter: 张洪胤 Assignee: 张洪胤 Add implementation for counting storage group. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3028) [cache] fix PartitionCache when there are ** in device path.
张洪胤 created IOTDB-3028: -- Summary: [cache] fix PartitionCache when there are ** in device path. Key: IOTDB-3028 URL: https://issues.apache.org/jira/browse/IOTDB-3028 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 Fix PartitionCache when there are ** in device path. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3014) [Rocksdb_based] OOM
[
https://issues.apache.org/jira/browse/IOTDB-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529157#comment-17529157
]
刘珍 commented on IOTDB-3014:
---
master_0428_b4611ed
OOM的问题，已解决，注册元数据成功，insert，{color:#DE350B}cli连接不上，jstack报错：{color}
jstack -l 2437 > 1_stack.out
2437: Unable to open socket file: target process not responding or HotSpot VM
not loaded
The -F option can be used when the target process is not responding
本来想stack看下 下面的这几个cpu高的是什么线程：
!screenshot-1.png!
> [Rocksdb_based] OOM
>
>
> Key: IOTDB-3014
> URL: https://issues.apache.org/jira/browse/IOTDB-3014
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Jianyun Cheng
>Priority: Major
> Labels: pull-request-available
> Attachments: config.properties, log_all.log, screenshot-1.png
>
>
> master_0426_c22c140
> 8C32G ,堆内/堆外默认，schema_engine_mode=Rocksdb_based
> DEVICE_NUMBER=50
> SENSOR_NUMBER=20
> GROUP_NUMBER=10
> benchmark执行写入顺序数据（配置文件见附件）
> iotdb报错（日志见附件）：
> 2022-04-26 15:44:48,684 [pool-13-IoTDB-RPC-Client-10] WARN
> o.a.i.d.u.ErrorHandlingUtils:61 - Status code: INTERNAL_SERVER_ERROR(500),
> operation: executeNonQueryPlan failed
> {color:#DE350B}*java.lang.UnsupportedOperationException: null*{color}
> at
> org.apache.iotdb.db.metadata.schemaregion.rocksdb.mnode.RMeasurementMNode.getLastCacheContainer(RMeasurementMNode.java:145)
> at
> org.apache.iotdb.db.metadata.lastCache.LastCacheManager.updateLastCache(LastCacheManager.java:88)
> at
> org.apache.iotdb.db.metadata.LocalSchemaProcessor.updateLastCache(LocalSchemaProcessor.java:1163)
> at
> org.apache.iotdb.db.engine.storagegroup.DataRegion.tryToUpdateBatchInsertLastCache(DataRegion.java:1231)
> at
> org.apache.iotdb.db.engine.storagegroup.DataRegion.insertTablet(DataRegion.java:974)
> at
> org.apache.iotdb.db.engine.StorageEngine.insertTablet(StorageEngine.java:663)
> at
> org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:1905)
> at
> org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:309)
> at
> org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:53)
> at
> org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:2066)
> at
> org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.insertTablet(TSServiceImpl.java:1576)
> at
> org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablet.getResult(TSIService.java:3328)
> at
> org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablet.getResult(TSIService.java:3308)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2022-04-26 15:46:10,208 [pool-13-IoTDB-RPC-Client-7] ERROR
> o.a.i.c.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread
> pool-13-IoTDB-RPC-Client-7-36
> {color:#DE350B}*java.lang.OutOfMemoryError: GC overhead limit exceeded*{color}
> at org.rocksdb.RocksDB.keyMayExistFoundValue(Native Method)
> at org.rocksdb.RocksDB.keyMayExist(RocksDB.java:2800)
> at org.rocksdb.RocksDB.keyMayExist(RocksDB.java:2663)
> at org.rocksdb.RocksDB.keyMayExist(RocksDB.java:2601)
> at org.rocksdb.RocksDB.keyMayExist(RocksDB.java:2570)
> at
> org.apache.iotdb.db.metadata.schemaregion.rocksdb.RSchemaReadWriteHandler.keyExist(RSchemaReadWriteHandler.java:299)
> at
> org.apache.iotdb.db.metadata.schemaregion.rocksdb.RSchemaReadWriteHandler.keyExistByType(RSchemaReadWriteHandler.java:254)
> at
> org.apache.iotdb.db.metadata.schemaregion.rocksdb.RSchemaRegion.getMeasurementMNode(RSchemaRegion.java:1229)
> at
> org.apache.iotdb.db.metadata.schemaregion.rocksdb.RSchemaRegion.getSeriesSchemasAndReadLockDevice(RSchemaRegion.java:1742)
> at
> org.apache.iotdb.db.metadata.LocalSchemaProcessor.getSeriesSchemasAndReadLockDevice(LocalSchemaProcessor.java:1259)
> at
[jira] [Created] (IOTDB-3031) [ClusterPartitionFetcher] Standardization of exception message
张洪胤 created IOTDB-3031: -- Summary: [ClusterPartitionFetcher] Standardization of exception message Key: IOTDB-3031 URL: https://issues.apache.org/jira/browse/IOTDB-3031 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 The message of exception in ClusterPartitionFetcher need to be check. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3036) [Cache] Fix SG auto create when use cache.
张洪胤 created IOTDB-3036: -- Summary: [Cache] Fix SG auto create when use cache. Key: IOTDB-3036 URL: https://issues.apache.org/jira/browse/IOTDB-3036 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3040) [ TTL ] Expired tsfiles with a TTL of more than 10 hours are not deleted
刘珍 created IOTDB-3040:
-
Summary: [ TTL ] Expired tsfiles with a TTL of more than 10 hours
are not deleted
Key: IOTDB-3040
URL: https://issues.apache.org/jira/browse/IOTDB-3040
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Server
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Jialin Qiao
Attachments: config.properties, iotdb-engine.properties, iotdb-env.sh,
logback.xml
master_0427_b633df5
问题描述：
长测配置，设置TTL为24小时，
grep ttl logs/log_all.log |tail -1
2022-04-29 08:57:07,404 [pool-11-IoTDB-TTL-Check-1] INFO
o.a.i.d.e.s.DataRegion:1658 - Removed a file
/data/iotdb_data/data/sequence/root.test.g_4/0/0/1651092583331-4021-1-0.tsfile{color:#DE350B}
before Sat Sep 01 19:33:53 CST 2018 by ttl{color} (115392194000ms)
./tools/tsfileToolSet/print-tsfile-resource-files.sh
解析sg中第1个resource文件，查看时间戳范围，最大时间戳：
device root.test.g_0.d_350, start time 1535749930200
(2018-09-01T05:12:10.200+08:00[Asia/Shanghai]), {color:#DE350B}*end time
1535749948000 (2018-09-01T05:12:28+08:00[Asia/Shanghai])*{color}
有14个小时的过期文件没有被删除，磁盘空间无法释放。
1. 机器信息
私有云172.20.70.14/13 8C32G
数据库，bm配置文件见附件。
长测运行起来之后，24小时，设置TTL：
#!/bin/bash
#ttl默认大小24小时
ttl=8640
host=127.0.0.1
port=6667
function setTTLOfSg(){
input_new_ttl=$1
./sbin/start-cli.sh -h $host -p $port -e "set ttl to root.** $input_new_ttl"
}
#计算操作系统当前时间和iotdb 序列max_time的差值
function getDiffOfTime(){
# 操作系统当前值 秒
os_now=`date +%s`
max_time=`./sbin/start-cli.sh -h $host -p $port -e "show devices limit
1"|grep root|awk -F '|' '{print "./sbin/start-cli.sh -h " host " -p " port " -e
\"select max_time(s_0) from " $2 " \" "}' host=$host port=$port|sh|grep "|
"|awk -F '|' '{print $2}'`
diffTime=$os_now*=1000))-max_time))
echo $diffTime
}
# 2小时校正1次
#sleep 24h
for i in {1..4}
do
getDiffOfTime
new_ttl=$((ttl+diffTime))
echo $new_ttl
setTTLOfSg $new_ttl
sleep 2h
done
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3044) [ concurrent delete sg ] There are some timeseries that cannot be deleted
刘珍 created IOTDB-3044:
-
Summary: [ concurrent delete sg ] There are some timeseries that
cannot be deleted
Key: IOTDB-3044
URL: https://issues.apache.org/jira/browse/IOTDB-3044
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Schema Manager
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Jialin Qiao
Attachments: image-2022-04-29-13-45-06-033.png
master_0428_ca4f3cf
问题描述：
50sg，500 dev, 20 sensor/dev
show devices的结果输出到dev_name.txt
顺序读取dev_name.txt中的设备名称，一个设备上1次delete 1个ts，10用户并发执行delete。
有的ts delete不掉。
日志中记录delete ts的操作，show timeseries还可以看到：
!image-2022-04-29-13-45-06-033.png!
测试流程
1. 192.168.10.68 机器
benchmark生成元数据。
2. 获取devices （放在${iotdb_dir}）
执行脚本
get_dev_name.sh
#!/bin/bash
./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
dev_name.txt
3. 启动并行delete （放在${iotdb_dir}）
cat del_10ts.sh
#!/bin/bash
function delete_ts()
{
del_ts=$1
dev_name=$2
exp_count=$3
count_ts ${exp_count} ${dev_name}
./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
let exp_count--
count_ts ${exp_count} ${dev_name}
}
function count_ts()
{
exp_c_ts=$1
dev_name=$2
cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
"4,4p"|awk -F '|' '{print $2}'`
if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
echo "ok"
else
echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
fi
}
function execute()
{
in_file=$1
i=$2
exp_count=$3
n=0
cat ${in_file} | while read line
do
del_ts="${line}.s_${i}"
dev_name="${line}.*";
delete_ts ${del_ts} ${dev_name} ${exp_count} &
let i++
let n++
b=$(( $n % 10 ))
if [ "$b" -eq "0" ];then
wait
fi
done
wait
}
# 循环几轮
count_ts_base=20
for loop in {0..99}
do
idx=$(( $loop * 2000 ))
execute "dev_name.txt" ${idx} ${count_ts_base}
let count_ts_base--
done
idx=19
execute "dev_name.txt" ${idx} ${count_ts_base}
不符合预期的结果会打印到fail.log
根据fail中记录的 ts名称，查看日志，有对应的delete ，show 可以看到没有被delete掉。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3045) [ schema recovery ] The query result contains some timeseries that have been deleted
刘珍 created IOTDB-3045:
-
Summary: [ schema recovery ] The query result contains some
timeseries that have been deleted
Key: IOTDB-3045
URL: https://issues.apache.org/jira/browse/IOTDB-3045
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Schema Manager
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Jialin Qiao
Attachments: config.properties, d1.log
master_0428_ca4f3cf
问题描述：
50 sg，100 dev，20 sensors/dev, 共2000万序列。
100个设备名称记录在文件中，
遍历文件，10个用户并发 delete ts，1个设备上delete 1条序列。
重复10次，也就是这100个dev，每个dev共delete 10个ts。
delete 后，执行count timeseries dev.* ；结果符合预期。
备份数据。
{color:#DE350B}重启iotdb，恢复元数据后，再查询 100个dev的序列数，有14个设备的序列数都是20万，
期望是10，delete的ts，又查询到了。{color}
详细测试流程：
1. 私有云 192.168.130.2
2. benchmark 配置见附件
3. 获取dev 名称，记录到文件dev_name.txt
cat get_dev_name.sh
#!/bin/bash
./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
dev_name.txt
4. 并发delete
cat del_1_ts.sh
#!/bin/bash
function delete_ts()
{
del_ts=$1
dev_name=$2
exp_count=$3
count_ts ${exp_count} ${dev_name}
./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
let exp_count--
count_ts ${exp_count} ${dev_name}
}
function count_ts()
{
exp_c_ts=$1
dev_name=$2
cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
"4,4p"|awk -F '|' '{print $2}'`
if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
echo "ok"
else
echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
fi
}
function execute()
{
in_file=$1
i=$2
exp_count=$3
n=0
cat ${in_file} | while read line
do
del_ts="${line}.s_${i}"
dev_name="${line}.*";
delete_ts ${del_ts} ${dev_name} ${exp_count} &
let i++
let n++
b=$(( $n % 10 ))
if [ "$b" -eq "0" ];then
wait
fi
done
wait
}
# 循环几轮
count_ts_base=20
for loop in {1..10}
do
execute "dev_name.txt" ${loop} ${count_ts_base}
let count_ts_base--
done
并发delete脚本的日志文件见附件。
执行count_ts.sh，查看每个dev的count timeseries，是否都是10。
有14个dev的结果是20，不符合预期结果。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3077) Doc of tsFile version generation requirement scenario
周沛辰 created IOTDB-3077: -- Summary: Doc of tsFile version generation requirement scenario Key: IOTDB-3077 URL: https://issues.apache.org/jira/browse/IOTDB-3077 Project: Apache IoTDB Issue Type: Task Reporter: 周沛辰 Assignee: 周沛辰 # When the target file generated by the compaction is too large, it needs to be divided into multiple target files, and then the conflict of the version number of the new file will be encountered. # When loading tsfile externally, it may appear that its version number conflicts with the version number of the existing file in the system. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3077) Doc of tsFile version generation requirement scenario
[ https://issues.apache.org/jira/browse/IOTDB-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531021#comment-17531021 ] 周沛辰 commented on IOTDB-3077: See doc in feishu: https://y8dp9fjm8f.feishu.cn/docs/doccnkVkvy5fYFkZWaPFSWiMj1c?from=from_copylink. > Doc of tsFile version generation requirement scenario > -- > > Key: IOTDB-3077 > URL: https://issues.apache.org/jira/browse/IOTDB-3077 > Project: Apache IoTDB > Issue Type: Task >Reporter: 周沛辰 >Assignee: 周沛辰 >Priority: Major > Original Estimate: 16h > Remaining Estimate: 16h > > # When the target file generated by the compaction is too large, it needs to > be divided into multiple target files, and then the conflict of the version > number of the new file will be encountered. > # When loading tsfile externally, it may appear that its version number > conflicts with the version number of the existing file in the system. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3096) [Auth] fix default config of user folder and role folder
张洪胤 created IOTDB-3096: -- Summary: [Auth] fix default config of user folder and role folder Key: IOTDB-3096 URL: https://issues.apache.org/jira/browse/IOTDB-3096 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 When use default config of common config, there are some errors of the path of user and role folder, so we need to first init common config according to iotdb config. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-2797) [privilege] root.** doesn't work
[ https://issues.apache.org/jira/browse/IOTDB-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532653#comment-17532653 ] 刘珍 commented on IOTDB-2797: --- 不应该是支持 root.** 吗 ? > [privilege] root.** doesn't work > -- > > Key: IOTDB-2797 > URL: https://issues.apache.org/jira/browse/IOTDB-2797 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Server >Affects Versions: 0.13.0 >Reporter: 刘珍 >Assignee: Wei Liu >Priority: Major > Labels: pull-request-available > Attachments: image-2022-03-24-14-40-26-163.png > > > 0.13.0 rc1 > 给用户root.**路径对应的权限，无效。 > delete storage group root.**; > drop user lily_create_trig; > create user lily_create_trig '123456'; > set storage group to root.sg1; > CREATE TIMESERIES root.sg1.dev0.s_1 WITH DATATYPE=INT32, ENCODING=GORILLA; > set storage group to root.sg2; > CREATE TIMESERIES root.sg2.dev0.s_1 WITH DATATYPE=INT32, ENCODING=GORILLA; > CREATE TIMESERIES root.sg1.dev1.s_1 WITH DATATYPE=INT32, ENCODING=GORILLA; > grant user lily_create_trig privileges CREATE_TRIGGER on root.**; > list privileges user lily_create_trig on root.**; > list privileges user lily_create_trig on root.sg1; > 存储组root.sg1上没有权限： > !image-2022-03-24-14-40-26-163.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-2797) [privilege] root.** doesn't work
[ https://issues.apache.org/jira/browse/IOTDB-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532733#comment-17532733 ] 刘珍 commented on IOTDB-2797: --- 跟乔老师讨论了一下：root.** 和 root应该都支持，具体也可以问下乔老师。 > [privilege] root.** doesn't work > -- > > Key: IOTDB-2797 > URL: https://issues.apache.org/jira/browse/IOTDB-2797 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Server >Affects Versions: 0.13.0 >Reporter: 刘珍 >Assignee: Wei Liu >Priority: Major > Labels: pull-request-available > Attachments: image-2022-03-24-14-40-26-163.png > > > 0.13.0 rc1 > 给用户root.**路径对应的权限，无效。 > delete storage group root.**; > drop user lily_create_trig; > create user lily_create_trig '123456'; > set storage group to root.sg1; > CREATE TIMESERIES root.sg1.dev0.s_1 WITH DATATYPE=INT32, ENCODING=GORILLA; > set storage group to root.sg2; > CREATE TIMESERIES root.sg2.dev0.s_1 WITH DATATYPE=INT32, ENCODING=GORILLA; > CREATE TIMESERIES root.sg1.dev1.s_1 WITH DATATYPE=INT32, ENCODING=GORILLA; > grant user lily_create_trig privileges CREATE_TRIGGER on root.**; > list privileges user lily_create_trig on root.**; > list privileges user lily_create_trig on root.sg1; > 存储组root.sg1上没有权限： > !image-2022-03-24-14-40-26-163.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3105) [ 0.13.0 rc1 ] File handle not released
刘珍 created IOTDB-3105:
-
Summary: [ 0.13.0 rc1 ] File handle not released
Key: IOTDB-3105
URL: https://issues.apache.org/jira/browse/IOTDB-3105
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Server
Affects Versions: 0.13.0
Reporter: 刘珍
Assignee: Jialin Qiao
Attachments: auto_set_ttl.sh, config.properties,
image-2022-05-06-17-38-19-399.png, image-2022-05-06-17-39-36-364.png,
iotdb-engine.properties, iotdb-env.sh
0.13.0 rc1
Long test configuration
查询耗时出现急剧上升，有查询超时，lsof -p ${iotdb_pid} 持有文件句柄多：
!image-2022-05-06-17-39-36-364.png!
查询耗时性能曲线：
!image-2022-05-06-17-38-19-399.png!
测试机器信息：
私有云8C32G * 2：172.20.70.28（iotdb）/30（benchmark）
数据库配置文件，benchmark配置文件见附件。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (IOTDB-3020) Create schema is slow
[ https://issues.apache.org/jira/browse/IOTDB-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533227#comment-17533227 ] 刘珍 commented on IOTDB-3020: --- master_0505_38473d1 mlog_buffer_size=10485760 schema_engine_mode=Schema_File 其余参数不变，机器同issue中描述（fit 1/2 8C32G） benchmark 工具使用0428_dce7a15（支持多用户并发创建schema），10用户并发，10sg ，50dev，20万sensors/dev , 共1000万序列，测试结果，4.5小时注册完成，此问题没解决: !screenshot-2.png! > Create schema is slow > - > > Key: IOTDB-3020 > URL: https://issues.apache.org/jira/browse/IOTDB-3020 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Schema Manager >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: ZhaoXin >Priority: Major > Labels: pull-request-available > Attachments: image-2022-04-27-15-24-15-249.png, screenshot-1.png, > screenshot-2.png > > > master_0427_4bf3dbb > 问题描述： > sensor_number 大，create schema会非常慢： > !screenshot-1.png! > 8C32G/schema_engine_mode=Schema_File > fit机器192.168.130.1/2 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3020) Create schema is slow
[
https://issues.apache.org/jira/browse/IOTDB-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533231#comment-17533231
]
刘珍 commented on IOTDB-3020:
---
master_0505_38473d1
schema_engine_mode=Schema_File
其余配置参数不更改。
fit机器（16C32G，bm和iotdb在同一台机器 ip-5），benchmark（20220322版本）是单线程创建schema，
{color:#DE350B}1sg{color}，2000dev，1万序列/dev
，共{color:#DE350B}2000万{color}序列，{color:#57D9A3}*耗时9分钟16秒。*{color}
!screenshot-3.png!
> Create schema is slow
> -
>
> Key: IOTDB-3020
> URL: https://issues.apache.org/jira/browse/IOTDB-3020
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: ZhaoXin
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-04-27-15-24-15-249.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> master_0427_4bf3dbb
> 问题描述：
> sensor_number 大，create schema会非常慢：
> !screenshot-1.png!
> 8C32G/schema_engine_mode=Schema_File
> fit机器192.168.130.1/2
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3111) [ metadata ] 10 users register metadata concurrently, but iotdb starts to register after more than 20 minutes
刘珍 created IOTDB-3111: - Summary: [ metadata ] 10 users register metadata concurrently, but iotdb starts to register after more than 20 minutes Key: IOTDB-3111 URL: https://issues.apache.org/jira/browse/IOTDB-3111 Project: Apache IoTDB Issue Type: Bug Components: Core/Schema Manager Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Jialin Qiao Attachments: config.properties, config.properties.10client-10sg-50dev-20wSensors, image-2022-05-07-17-37-36-582.png master_0505_38473d1 Problem Description： 10 users register metadata concurrently, but iotdb starts to register after more than 20 minutes (即：10用户并发注册元数据，iotdb 20多分钟才开始注册 , log输出set storage group to.. create storage group schema folder..） 机器信息： fit 192.168.130.1（iotdb 8C32G）/ 2 (bm) iotdb配置参数： mlog_buffer_size=10485760 schema_engine_mode=Schema_File 其余参数默认。 bm配置参数见附件（2个配置都可以复现）。 日志（24分钟后才开始注册，rel/0.13没这个问题，可以排除benchmark的问题）： !image-2022-05-07-17-37-36-582.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-3111) [ benchmark ] The implementation of creating templates is slow
[
https://issues.apache.org/jira/browse/IOTDB-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
张洪胤 reassigned IOTDB-3111:
--
Assignee: 张洪胤 (was: Jialin Qiao)
> [ benchmark ] The implementation of creating templates is slow
> --
>
> Key: IOTDB-3111
> URL: https://issues.apache.org/jira/browse/IOTDB-3111
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Benchmark
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: 张洪胤
>Priority: Minor
> Attachments: config.properties,
> config.properties.10client-10sg-50dev-20wSensors,
> image-2022-05-07-17-37-36-582.png
>
>
> 问题描述：
> benchmark创建template 慢，包含20万序列的template 耗时24分钟。
> bm配置参数见附件（需再{color:#DE350B}设置一下TEMPLATE=true{color}）。
> 日志（24分钟后才开始set storage group ...因为这期间在创建template）
> !image-2022-05-07-17-37-36-582.png!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (IOTDB-3020) Create schema is slow
[ https://issues.apache.org/jira/browse/IOTDB-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533442#comment-17533442 ] 刘珍 commented on IOTDB-3020: --- 现在看起来是sensor_number大benchmark准备数据慢，压力上不去的问题，bm可以把准备数据的时间排除掉。 > Create schema is slow > - > > Key: IOTDB-3020 > URL: https://issues.apache.org/jira/browse/IOTDB-3020 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Schema Manager >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: ZhaoXin >Priority: Major > Labels: pull-request-available > Attachments: image-2022-04-27-15-24-15-249.png, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > master_0427_4bf3dbb > 问题描述： > sensor_number 大，create schema会非常慢： > !screenshot-1.png! > 8C32G/schema_engine_mode=Schema_File > fit机器192.168.130.1/2 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3119) Add some optional parameters of storage group related sql.
张洪胤 created IOTDB-3119: -- Summary: Add some optional parameters of storage group related sql. Key: IOTDB-3119 URL: https://issues.apache.org/jira/browse/IOTDB-3119 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 set storage group statement are extended: setStorageGroup : SET STORAGE GROUP TO prefixPath (WITH sgAttributeClause (COMMA sgAttributeClause)*)? ; sgAttributeClause : (TTL | SCHEMA_REPLICATION_FACTOR | DATA_REPLICATION_FACTOR | TIME_PARTITION_INTERVAL) '=' INTEGER_LITERAL ; when you use show storage group, add ttl, dataReplicationFactor, schemaReplicationFactor and timePartitionInterval into result. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3121) [ ICachedMNodeContainer.java ] MNodeContainers$EmptyContainer cannot be cast to org.apache.iotdb.db.metadata.mtree.store.disk.ICachedMNodeContainer
刘珍 created IOTDB-3121:
-
Summary: [ ICachedMNodeContainer.java ]
MNodeContainers$EmptyContainer cannot be cast to
org.apache.iotdb.db.metadata.mtree.store.disk.ICachedMNodeContainer
Key: IOTDB-3121
URL: https://issues.apache.org/jira/browse/IOTDB-3121
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Schema Manager
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: ZhaoXin
Attachments: config.properties.1, config.properties.2
master_0509_bc6c4af
问题描述：
2用户并发创建schema ：
2022-05-09 11:20:55,548 [pool-16-IoTDB-RPC-Client-4] WARN
o.a.i.d.u.ErrorHandlingUtils:61 - Status code: INTERNAL_SERVER_ERROR(500),
operation: executeNonQueryPlan failed
java.lang.ClassCastException:
org.apache.iotdb.db.metadata.mnode.container.MNodeContainers$EmptyContainer
cannot be cast to
org.apache.iotdb.db.metadata.mtree.store.disk.ICachedMNodeContainer
at
org.apache.iotdb.db.metadata.mtree.store.disk.ICachedMNodeContainer.getBelongedContainer(ICachedMNodeContainer.java:78)
at
org.apache.iotdb.db.metadata.mtree.store.disk.cache.CacheManager.updateCacheStatusAfterAppend(CacheManager.java:124)
at
org.apache.iotdb.db.metadata.mtree.store.CachedMTreeStore.addChild(CachedMTreeStore.java:201)
at
org.apache.iotdb.db.metadata.mtree.MTreeBelowSGCachedImpl.createTimeseriesWithPinnedReturn(MTreeBelowSGCachedImpl.java:225)
at
org.apache.iotdb.db.metadata.schemaregion.SchemaRegionSchemaFileImpl.createTimeseries(SchemaRegionSchemaFileImpl.java:466)
at
org.apache.iotdb.db.metadata.LocalSchemaProcessor.createTimeseries(LocalSchemaProcessor.java:290)
at
org.apache.iotdb.db.metadata.LocalSchemaProcessor.createTimeseries(LocalSchemaProcessor.java:285)
at
org.apache.iotdb.db.qp.executor.PlanExecutor.createTimeSeries(PlanExecutor.java:1995)
at
org.apache.iotdb.db.qp.executor.PlanExecutor.createMultiTimeSeries(PlanExecutor.java:2034)
at
org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:339)
at
org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:53)
at
org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:2066)
at
org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.createMultiTimeseries(TSServiceImpl.java:1858)
at
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$createMultiTimeseries.getResult(TSIService.java:3203)
at
org.apache.iotdb.service.rpc.thrift.TSIService$Processor$createMultiTimeseries.getResult(TSIService.java:3183)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
复现流程
1. 机器
fit 192.168.130.1 (8C32G) /home/iotdb/db/master_0509_bc6c4af
MAX_HEAP_SIZE="20G"
MAX_DIRECT_MEMORY_SIZE="6G"
schema_engine_mode=Schema_File
fit 192.168.130.2 /home/benchmark/bm_20220322
2. bm 配置文件
放在/home/benchmark/bm_20220322/conf
3. 启动iotdb
4. 运行bm
/home/benchmark/bm_20220322下运行run.sh
cat run.sh
#!/bin/bash
time1=`date +%s`
cp -rp conf/config.properties.1 conf/config.properties
./benchmark.sh > 1.out &
cp -rp conf/config.properties.2 conf/config.properties
./benchmark.sh > 2.out &
wait
time2=`date +%s`
time=$(($time2-$time1))
echo "bing fa run : ${time} seconds." >> time_res.txt
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Assigned] (IOTDB-3123) [Metrics]IoTDB Reporter is not work at a certain point in time while Prometheus Reporter is working fine.
[ https://issues.apache.org/jira/browse/IOTDB-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张洪胤 reassigned IOTDB-3123: -- Assignee: 张洪胤 > [Metrics]IoTDB Reporter is not work at a certain point in time while > Prometheus Reporter is working fine. > - > > Key: IOTDB-3123 > URL: https://issues.apache.org/jira/browse/IOTDB-3123 > Project: Apache IoTDB > Issue Type: Bug >Reporter: ErickinYae >Assignee: 张洪胤 >Priority: Major > > Writing metric data to the IoTDB table stopped at some point in time. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Reopened] (IOTDB-3045) [ schema recovery ] The query result contains some timeseries that have been deleted
[
https://issues.apache.org/jira/browse/IOTDB-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
刘珍 reopened IOTDB-3045:
---
master_0508_be385cd
100sg，500dev，20w序列/dev，共1亿序列，每个序列有10个点的数据。
每个dev delete 101个ts，重启恢复，部分dev的序列数是20万，期望是199899。
机器 ：192.168.10.68
MAX_HEAP_SIZE="192G"
MAX_DIRECT_MEMORY_SIZE="32G"
schema_engine_mode=Schema_File
benchmark配置：
GROUP_NUMBER=100
DEVICE_NUMBER=500
SENSOR_NUMBER=20
CLIENT_NUMBER=50
BATCH_SIZE_PER_WRITE=1
LOOP=10
脚本${iotdb_dir}下：
先获取所有dev的信息
cat get_dev_name.sh
#!/bin/bash
./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
dev_name.txt
再执行delete ts（10用户并发）
cat get_dev_name.sh
#!/bin/bash
./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
dev_name.txt
liuzhen@fit-68:/data/liuzhen_test/master_0508_be385cd$ cat del_ts.sh
#!/bin/bash
function delete_ts()
{
del_ts=$1
dev_name=$2
exp_count=$3
count_ts ${exp_count} ${dev_name}
./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
let exp_count--
count_ts ${exp_count} ${dev_name}
}
function count_ts()
{
exp_c_ts=$1
dev_name=$2
cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
"4,4p"|awk -F '|' '{print $2}'`
if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
echo "ok"
else
echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
fi
}
function execute()
{
in_file=$1
i=$2
exp_count=$3
n=0
cat ${in_file} | while read line
do
del_ts="${line}.s_${i}"
dev_name="${line}.*";
delete_ts ${del_ts} ${dev_name} ${exp_count} &
let i++
let n++
b=$(( $n % 10 ))
if [ "$b" -eq "0" ];then
wait
fi
done
wait
}
function execute_ts_maxID()
{
in_file=$1
maxID=$2
exp_count=$3
n=0
cat ${in_file} | while read line
do
del_ts="${line}.s_${maxID}"
dev_name="${line}.*";
delete_ts ${del_ts} ${dev_name} ${exp_count} &
let n++
b=$(( $n % 10 ))
if [ "$b" -eq "0" ];then
wait
fi
done
wait
}
# 循环几轮
count_ts_base=20
for loop in {0..99}
do
idx=$(( $loop * 2000 ))
execute "dev_name.txt" ${idx} ${count_ts_base}
let count_ts_base--
done
idx=19
execute_ts_maxID "dev_name.txt" ${idx} ${count_ts_base}
执行完后，再查询一遍所有dev下的count timeseries , 均是199899。表现正确。
停iotdb。
备份数据。
启动iotdb，查询一遍所有dev下的count timeseries ,部分结果是200000，delete成功的ts又查询到了。
> [ schema recovery ] The query result contains some timeseries that have been
> deleted
>
>
> Key: IOTDB-3045
> URL: https://issues.apache.org/jira/browse/IOTDB-3045
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Jialin Qiao
>Priority: Major
> Attachments: config.properties, d1.log
>
>
> master_0428_ca4f3cf
> 问题描述：
> 50 sg，100 dev，20 sensors/dev, 共2000万序列。
> 100个设备名称记录在文件中，
> 遍历文件，10个用户并发 delete ts，1个设备上delete 1条序列。
> 重复10次，也就是这100个dev，每个dev共delete 10个ts。
> delete 后，执行count timeseries dev.* ；结果符合预期。
> 备份数据。
> {color:#DE350B}重启iotdb，恢复元数据后，再查询 100个dev的序列数，有14个设备的序列数都是20万，
> 期望是10，delete的ts，又查询到了。{color}
> 详细测试流程：
> 1. 私有云 172.20.70.2(iotdb)/16(bm) 8C32G
> iotdb参数
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> schema_engine_mode=Schema_File
> 2. benchmark 配置见附件
> 3. 获取dev 名称，记录到文件dev_name.txt
> cat get_dev_name.sh
> #!/bin/bash
> ./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
> dev_name.txt
> 4. 并发delete
> cat del_1_ts.sh
> #!/bin/bash
> function delete_ts()
> {
>del_ts=$1
>dev_name=$2
>exp_count=$3
>count_ts ${exp_count} ${dev_name}
>./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
>let exp_count--
>count_ts ${exp_count} ${dev_name}
> }
> function count_ts()
> {
> exp_c_ts=$1
> dev_name=$2
> cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
> "4,4p"|awk -F '|' '{print $2}'`
> if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
> echo "ok"
> else
> echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
>
[jira] [Assigned] (IOTDB-3045) [ schema recovery ] The query result contains some timeseries that have been deleted
[
https://issues.apache.org/jira/browse/IOTDB-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
刘珍 reassigned IOTDB-3045:
-
Assignee: ZhaoXin (was: Jialin Qiao)
> [ schema recovery ] The query result contains some timeseries that have been
> deleted
>
>
> Key: IOTDB-3045
> URL: https://issues.apache.org/jira/browse/IOTDB-3045
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: ZhaoXin
>Priority: Major
> Attachments: config.properties, d1.log
>
>
> master_0428_ca4f3cf
> 问题描述：
> 50 sg，100 dev，20 sensors/dev, 共2000万序列。
> 100个设备名称记录在文件中，
> 遍历文件，10个用户并发 delete ts，1个设备上delete 1条序列。
> 重复10次，也就是这100个dev，每个dev共delete 10个ts。
> delete 后，执行count timeseries dev.* ；结果符合预期。
> 备份数据。
> {color:#DE350B}重启iotdb，恢复元数据后，再查询 100个dev的序列数，有14个设备的序列数都是20万，
> 期望是10，delete的ts，又查询到了。{color}
> 详细测试流程：
> 1. 私有云 172.20.70.2(iotdb)/16(bm) 8C32G
> iotdb参数
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> schema_engine_mode=Schema_File
> 2. benchmark 配置见附件
> 3. 获取dev 名称，记录到文件dev_name.txt
> cat get_dev_name.sh
> #!/bin/bash
> ./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
> dev_name.txt
> 4. 并发delete
> cat del_1_ts.sh
> #!/bin/bash
> function delete_ts()
> {
>del_ts=$1
>dev_name=$2
>exp_count=$3
>count_ts ${exp_count} ${dev_name}
>./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
>let exp_count--
>count_ts ${exp_count} ${dev_name}
> }
> function count_ts()
> {
> exp_c_ts=$1
> dev_name=$2
> cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
> "4,4p"|awk -F '|' '{print $2}'`
> if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
> echo "ok"
> else
> echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
> fi
> }
> function execute()
> {
> in_file=$1
> i=$2
> exp_count=$3
> n=0
> cat ${in_file} | while read line
> do
>del_ts="${line}.s_${i}"
>dev_name="${line}.*";
>delete_ts ${del_ts} ${dev_name} ${exp_count} &
>let i++
>let n++
> b=$(( $n % 10 ))
>if [ "$b" -eq "0" ];then
> wait
>fi
> done
> wait
> }
> # 循环几轮
> count_ts_base=20
> for loop in {1..10}
> do
> execute "dev_name.txt" ${loop} ${count_ts_base}
> let count_ts_base--
> done
> 并发delete脚本的日志文件见附件。
> 执行count_ts.sh，查看每个dev的count timeseries，是否都是10。
> 有14个dev的结果是20，不符合预期结果。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (IOTDB-1239) Incorrect datetime value can be inserted into iotdb successfully.
[
https://issues.apache.org/jira/browse/IOTDB-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534156#comment-17534156
]
刘珍 commented on IOTDB-1239:
---
!screenshot-1.png!
master_0510_19872aa ， 已解决。时间戳值按合法处理。
> Incorrect datetime value can be inserted into iotdb successfully.
> -
>
> Key: IOTDB-1239
> URL: https://issues.apache.org/jira/browse/IOTDB-1239
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Client/CLI
>Reporter: 刘珍
>Assignee: Haonan Hou
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.14.0
>
> Attachments: image-2021-04-29-14-54-43-270.png, screenshot-1.png
>
>
> master branch.
> commit id 5fcff40f2299caeb0c8a9ae42f68bc18ecf8fee7 .
>
> SET STORAGE GROUP TO root.sg
> CREATE TIMESERIES root.sg.d1.s2 WITH DATATYPE=DOUBLE, ENCODING=PLAIN
> insert into root.sg.d1(time,s2 ) values (15741792,8.76);
> IoTDB> select * from root.sg.d1;
> +---+-+
> | Time|root.sg.d1.s2|
> +---+-+
> |{color:#FF}+{color}51853-10-03T00:00:{color:#FF}.0{color}+08:00|
> 8.76|
> +---+-+
> {color:#FF}should be give an error :{color}
> {color:#FF}Incorrect datetime value {color}
> IoTDB> select * from root.sg.d1 where time=51853-10-03T00:00:.0+08:00
> Msg: 401: Error occurred while parsing SQL to physical plan: line 1:36
> extraneous input '51853-10-03T00:00:' expecting \{NOW, TRUE, FALSE, '-', '.',
> 'NaN', INT, EXPONENT, DATETIME, DOUBLE_QUOTE_STRING_LITERAL,
> SINGLE_QUOTE_STRING_LITERAL}
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3137) IoTDB supports registering as a system service
刘珍 created IOTDB-3137: - Summary: IoTDB supports registering as a system service Key: IOTDB-3137 URL: https://issues.apache.org/jira/browse/IOTDB-3137 Project: Apache IoTDB Issue Type: Improvement Components: Core/Server Reporter: 刘珍 IoTDB supports registering as a system service -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-3137) IoTDB supports registering as a system service
[ https://issues.apache.org/jira/browse/IOTDB-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 刘珍 reassigned IOTDB-3137: - Assignee: Jialin Qiao > IoTDB supports registering as a system service > -- > > Key: IOTDB-3137 > URL: https://issues.apache.org/jira/browse/IOTDB-3137 > Project: Apache IoTDB > Issue Type: Improvement > Components: Core/Server >Reporter: 刘珍 >Assignee: Jialin Qiao >Priority: Minor > > IoTDB supports registering as a system service -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-2829) [template query performance] The query performance with templates is slower than that without templates
[
https://issues.apache.org/jira/browse/IOTDB-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534275#comment-17534275
]
刘珍 commented on IOTDB-2829:
---
!screenshot-2.png!
master_0510_19872aa 可以关闭。
> [template query performance] The query performance with templates is slower
> than that without templates
> ---
>
> Key: IOTDB-2829
> URL: https://issues.apache.org/jira/browse/IOTDB-2829
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Query
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: ZhaoXin
>Priority: Minor
> Attachments: no_template_luan-xu.config.properties, screenshot-1.png,
> screenshot-2.png, template-查询慢.zip, template_luan-xu.config.properties
>
>
> rel/0.13 60f90977a88f767c7ca86c6e10f1471bba262c2f
> 问题描述：
> 相同数据量，有乱序，带模板的查询性能比不带模板慢：
> !screenshot-1.png!
> 测试流程
> 1. 测试环境
> 192.168.10.68机器
> {color:#DE350B}*关合并*{color}
> 带template的数据备份在/data/benchmark/weekly_shell/rel_13_rc1_data_bk/data_3_luanxu
> 不带template的数据备份在/data/benchmark/weekly_shell/rel_13_rc1_data_bk/data_1_luanxu
> 数据备份大，不好copy到本地复现，也可以使用附件中的配置生成：
> 关合并，template_luan-xu.config.properties 配置生成带template的乱序数据。
> 关合并，no_template_luan-xu.config.properties配置生成不带template的乱序数据。
> 数据库路径：
> ${IOTDB_DIR}=/data/liuzhen_test/weekly_0127/13_rc1
> benchmark工具路径：
> ${benchmark_dir}=/data/benchmark/weekly_shell/benchmark_tool_20220322_v13
> 2. 测试流程
> for i in {Q1..Q10} do
> 清OS缓存，copy 数据备份到${IOTDB_DIR}，启动数据库。
> copy Qi..config.properties查询的配置文件到${benchmark_dir}
> 启动benchmark查询，记录结果
> done
> 查询语句的benchmark配置文件见附件：
> 对比结果，带template的查询耗时长。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (IOTDB-2258) [Aligned timeseries] export-csv.sh & import-csv.sh need to support aligned timeseries
[ https://issues.apache.org/jira/browse/IOTDB-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534656#comment-17534656 ] 刘珍 commented on IOTDB-2258: --- db_0507_9633491，已解决。 > [Aligned timeseries] export-csv.sh & import-csv.sh need to support aligned > timeseries > - > > Key: IOTDB-2258 > URL: https://issues.apache.org/jira/browse/IOTDB-2258 > Project: Apache IoTDB > Issue Type: New Feature > Components: Tools/Others >Affects Versions: 0.13.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Xuan Ronaldo >Priority: Minor > Labels: 0.13.0, pull-request-available > Fix For: 0.13.0 > > Attachments: image-2022-01-20-14-36-57-776.png > > > 如题。 > 1. 导出对齐时间序列的数据，再导入到干净的数据库，导入后的时间序列是非对齐的。 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3121) [ ICachedMNodeContainer.java ] MNodeContainers$EmptyContainer cannot be cast to org.apache.iotdb.db.metadata.mtree.store.disk.ICachedMNodeContainer
[
https://issues.apache.org/jira/browse/IOTDB-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534666#comment-17534666
]
刘珍 commented on IOTDB-3121:
---
PR
!screenshot-1.png!
> [ ICachedMNodeContainer.java ] MNodeContainers$EmptyContainer cannot be cast
> to org.apache.iotdb.db.metadata.mtree.store.disk.ICachedMNodeContainer
> ---
>
> Key: IOTDB-3121
> URL: https://issues.apache.org/jira/browse/IOTDB-3121
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: ZhaoXin
>Priority: Major
> Labels: pull-request-available
> Attachments: config.properties.1, config.properties.2,
> screenshot-1.png
>
>
> master_0509_bc6c4af
> 问题描述：
> 2用户并发创建schema ：
> 2022-05-09 11:20:55,548 [pool-16-IoTDB-RPC-Client-4] WARN
> o.a.i.d.u.ErrorHandlingUtils:61 - Status code: INTERNAL_SERVER_ERROR(500),
> operation: executeNonQueryPlan failed
> java.lang.ClassCastException:
> org.apache.iotdb.db.metadata.mnode.container.MNodeContainers$EmptyContainer
> cannot be cast to
> org.apache.iotdb.db.metadata.mtree.store.disk.ICachedMNodeContainer
> at
> org.apache.iotdb.db.metadata.mtree.store.disk.ICachedMNodeContainer.getBelongedContainer(ICachedMNodeContainer.java:78)
> at
> org.apache.iotdb.db.metadata.mtree.store.disk.cache.CacheManager.updateCacheStatusAfterAppend(CacheManager.java:124)
> at
> org.apache.iotdb.db.metadata.mtree.store.CachedMTreeStore.addChild(CachedMTreeStore.java:201)
> at
> org.apache.iotdb.db.metadata.mtree.MTreeBelowSGCachedImpl.createTimeseriesWithPinnedReturn(MTreeBelowSGCachedImpl.java:225)
> at
> org.apache.iotdb.db.metadata.schemaregion.SchemaRegionSchemaFileImpl.createTimeseries(SchemaRegionSchemaFileImpl.java:466)
> at
> org.apache.iotdb.db.metadata.LocalSchemaProcessor.createTimeseries(LocalSchemaProcessor.java:290)
> at
> org.apache.iotdb.db.metadata.LocalSchemaProcessor.createTimeseries(LocalSchemaProcessor.java:285)
> at
> org.apache.iotdb.db.qp.executor.PlanExecutor.createTimeSeries(PlanExecutor.java:1995)
> at
> org.apache.iotdb.db.qp.executor.PlanExecutor.createMultiTimeSeries(PlanExecutor.java:2034)
> at
> org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:339)
> at
> org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:53)
> at
> org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:2066)
> at
> org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.createMultiTimeseries(TSServiceImpl.java:1858)
> at
> org.apache.iotdb.service.rpc.thrift.TSIService$Processor$createMultiTimeseries.getResult(TSIService.java:3203)
> at
> org.apache.iotdb.service.rpc.thrift.TSIService$Processor$createMultiTimeseries.getResult(TSIService.java:3183)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 复现流程
> 1. 机器
> fit 192.168.130.1 (8C32G) /home/iotdb/db/master_0509_bc6c4af
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> schema_engine_mode=Schema_File
> fit 192.168.130.2 /home/benchmark/bm_20220322
> 2. bm 配置文件
> 放在/home/benchmark/bm_20220322/conf
> 3. 启动iotdb
> 4. 运行bm
> /home/benchmark/bm_20220322下运行run.sh
> cat run.sh
> #!/bin/bash
> time1=`date +%s`
> cp -rp conf/config.properties.1 conf/config.properties
> ./benchmark.sh > 1.out &
> cp -rp conf/config.properties.2 conf/config.properties
> ./benchmark.sh > 2.out &
> wait
> time2=`date +%s`
> time=$(($time2-$time1))
> echo "bing fa run : ${time} seconds." >> time_res.txt
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3156) [ compaction ] Uneven distribution of merge tasks per sg
刘珍 created IOTDB-3156:
-
Summary: [ compaction ] Uneven distribution of merge tasks per sg
Key: IOTDB-3156
URL: https://issues.apache.org/jira/browse/IOTDB-3156
Project: Apache IoTDB
Issue Type: Improvement
Components: Core/Compaction
Reporter: 刘珍
Assignee: Liuxuxin
Attachments: image-2022-05-11-15-45-33-156.png,
image-2022-05-11-15-45-38-071.png
问题描述 ：
0.13.0 rc1 每个sg的合并任务分布不均匀。
!image-2022-05-11-15-45-33-156.png!
!image-2022-05-11-15-45-38-071.png!
复现流程：
长测配置，顺序数据。
启动测试后，运行脚本：
cat get_sg_tsfile-count.sh
#!/bin/bash
sg_res=" "
function sg_tsfile()
{
tsfile_count=`ls -l
/data/iotdb_data/data/sequence/root.test.g_${1}/0/0/*.resource|wc -l`
sg_res="${sg_res} ${tsfile_count}"
}
while true
do
for i in {0..49}
do
sg_tsfile ${i}
done
time=`date "+%Y-%m-%d %H:%M:%S"`
echo "${time} ${sg_res} " >> sg_tsfile_res_0509.out
sleep 2h
sg_res=" "
done
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3157) [Benchmark] Fix the problem of schema module of benchmark
张洪胤 created IOTDB-3157: -- Summary: [Benchmark] Fix the problem of schema module of benchmark Key: IOTDB-3157 URL: https://issues.apache.org/jira/browse/IOTDB-3157 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 Now iotdb-benchmark use multi-thread to register schema, so we need to control all threads to avoid the repeat register of schema. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3158) [Metric] The problem when use iotdb-reporter
张洪胤 created IOTDB-3158: -- Summary: [Metric] The problem when use iotdb-reporter Key: IOTDB-3158 URL: https://issues.apache.org/jira/browse/IOTDB-3158 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 When use iotdb-reporter, metric module will stop to write metric into iotdb after a period of time. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3160) TsFile will be corrupted when flushing memtable appears OOM.
周沛辰 created IOTDB-3160: -- Summary: TsFile will be corrupted when flushing memtable appears OOM. Key: IOTDB-3160 URL: https://issues.apache.org/jira/browse/IOTDB-3160 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.13.0, master branch Reporter: 周沛辰 Attachments: image-2022-05-11-16-59-49-557.png, image-2022-05-11-17-08-56-755.png When flushing memtable appears OOM, the file will be corrupted with metadata only and cannot be repaired at the next restart. The result may be as following: When an exception is caught in the flushing memtable, the following logic will be executed to call the TsFileIOWriter's endfile, causing the file to have only the metadata area but no data area. After the next restart, memTable is considered to have flushed properly and WAL is not used to restore tsfile. !image-2022-05-11-16-59-49-557.png! !image-2022-05-11-17-08-56-755.png|width=1594,height=592! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3162) Split large target files in cross space compaction
周沛辰 created IOTDB-3162: -- Summary: Split large target files in cross space compaction Key: IOTDB-3162 URL: https://issues.apache.org/jira/browse/IOTDB-3162 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 Split large target files in cross space compaction with new rule of tsfile version. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3163) Refactor compaction memory estimator frame and add new estimator of RewriteCrossCompaction
周沛辰 created IOTDB-3163: -- Summary: Refactor compaction memory estimator frame and add new estimator of RewriteCrossCompaction Key: IOTDB-3163 URL: https://issues.apache.org/jira/browse/IOTDB-3163 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3164) Optimize memory allocation of the system and memory control of compaction
周沛辰 created IOTDB-3164: -- Summary: Optimize memory allocation of the system and memory control of compaction Key: IOTDB-3164 URL: https://issues.apache.org/jira/browse/IOTDB-3164 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3165) Implement data deletion in distributed system
周沛辰 created IOTDB-3165: -- Summary: Implement data deletion in distributed system Key: IOTDB-3165 URL: https://issues.apache.org/jira/browse/IOTDB-3165 Project: Apache IoTDB Issue Type: New Feature Reporter: 周沛辰 Assignee: 周沛辰 Implement data deletion in distributed system. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3045) [ schema recovery ] The query result contains some timeseries that have been deleted
[
https://issues.apache.org/jira/browse/IOTDB-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17535802#comment-17535802
]
刘珍 commented on IOTDB-3045:
---
重启前执行flush，表现是正确的（delete的 查询不到）。
> [ schema recovery ] The query result contains some timeseries that have been
> deleted
>
>
> Key: IOTDB-3045
> URL: https://issues.apache.org/jira/browse/IOTDB-3045
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: ZhaoXin
>Priority: Major
> Attachments: config.properties, d1.log
>
>
> master_0428_ca4f3cf
> 问题描述：
> 50 sg，100 dev，20 sensors/dev, 共2000万序列。
> 100个设备名称记录在文件中，
> 遍历文件，10个用户并发 delete ts，1个设备上delete 1条序列。
> 重复10次，也就是这100个dev，每个dev共delete 10个ts。
> delete 后，执行count timeseries dev.* ；结果符合预期。
> 备份数据。
> {color:#DE350B}重启iotdb，恢复元数据后，再查询 100个dev的序列数，有14个设备的序列数都是20万，
> 期望是10，delete的ts，又查询到了。{color}
> 详细测试流程：
> 1. 私有云 172.20.70.2(iotdb)/16(bm) 8C32G
> iotdb参数
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> schema_engine_mode=Schema_File
> 2. benchmark 配置见附件
> 3. 获取dev 名称，记录到文件dev_name.txt
> cat get_dev_name.sh
> #!/bin/bash
> ./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
> dev_name.txt
> 4. 并发delete
> cat del_1_ts.sh
> #!/bin/bash
> function delete_ts()
> {
>del_ts=$1
>dev_name=$2
>exp_count=$3
>count_ts ${exp_count} ${dev_name}
>./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
>let exp_count--
>count_ts ${exp_count} ${dev_name}
> }
> function count_ts()
> {
> exp_c_ts=$1
> dev_name=$2
> cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
> "4,4p"|awk -F '|' '{print $2}'`
> if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
> echo "ok"
> else
> echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
> fi
> }
> function execute()
> {
> in_file=$1
> i=$2
> exp_count=$3
> n=0
> cat ${in_file} | while read line
> do
>del_ts="${line}.s_${i}"
>dev_name="${line}.*";
>delete_ts ${del_ts} ${dev_name} ${exp_count} &
>let i++
>let n++
> b=$(( $n % 10 ))
>if [ "$b" -eq "0" ];then
> wait
>fi
> done
> wait
> }
> # 循环几轮
> count_ts_base=20
> for loop in {1..10}
> do
> execute "dev_name.txt" ${loop} ${count_ts_base}
> let count_ts_base--
> done
> 并发delete脚本的日志文件见附件。
> 执行count_ts.sh，查看每个dev的count timeseries，是否都是10。
> 有14个dev的结果是20，不符合预期结果。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3173) Support Between expression in Transform Operator
李伟豪 created IOTDB-3173: -- Summary: Support Between expression in Transform Operator Key: IOTDB-3173 URL: https://issues.apache.org/jira/browse/IOTDB-3173 Project: Apache IoTDB Issue Type: Task Reporter: 李伟豪 Assignee: 李伟豪 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3174) Support Between expression in Transform Operator
李伟豪 created IOTDB-3174: -- Summary: Support Between expression in Transform Operator Key: IOTDB-3174 URL: https://issues.apache.org/jira/browse/IOTDB-3174 Project: Apache IoTDB Issue Type: Task Reporter: 李伟豪 Assignee: 李伟豪 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3175) Unable to find file when creating user with AuthorInfo
任宇华 created IOTDB-3175: -- Summary: Unable to find file when creating user with AuthorInfo Key: IOTDB-3175 URL: https://issues.apache.org/jira/browse/IOTDB-3175 Project: Apache IoTDB Issue Type: Bug Reporter: 任宇华 Attachments: image-2022-05-13-14-05-57-117.png !image-2022-05-13-14-05-57-117.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-3175) Unable to find file when creating user with AuthorInfo
[ https://issues.apache.org/jira/browse/IOTDB-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 任宇华 reassigned IOTDB-3175: -- Assignee: 任宇华 > Unable to find file when creating user with AuthorInfo > -- > > Key: IOTDB-3175 > URL: https://issues.apache.org/jira/browse/IOTDB-3175 > Project: Apache IoTDB > Issue Type: Bug >Reporter: 任宇华 >Assignee: 任宇华 >Priority: Major > Attachments: image-2022-05-13-14-05-57-117.png > > > > !image-2022-05-13-14-05-57-117.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3175) Unable to find file when creating user with AuthorInfo
[ https://issues.apache.org/jira/browse/IOTDB-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536441#comment-17536441 ] 任宇华 commented on IOTDB-3175: https://github.com/apache/iotdb/pull/5867 > Unable to find file when creating user with AuthorInfo > -- > > Key: IOTDB-3175 > URL: https://issues.apache.org/jira/browse/IOTDB-3175 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Authority >Affects Versions: master branch >Reporter: 任宇华 >Assignee: 任宇华 >Priority: Major > Fix For: master branch > > Attachments: image-2022-05-13-14-05-57-117.png > > > When calling the AuthorInfo interface to create a user, in the ubuntu > environment, the folder will not be created > !image-2022-05-13-14-05-57-117.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3176) [ delete dev.** ] Improve delete device performance
刘珍 created IOTDB-3176: - Summary: [ delete dev.** ] Improve delete device performance Key: IOTDB-3176 URL: https://issues.apache.org/jira/browse/IOTDB-3176 Project: Apache IoTDB Issue Type: Improvement Components: Core/Schema Manager Reporter: 刘珍 Assignee: ZhaoXin Attachments: del_dev_test.sh, image-2022-05-13-16-26-32-480.png 改进描述： delete timeseries [root.test.g_42.test.schema_t.d_242.**] DDL 操作耗时近14分钟，可考虑优化。 如下日志 , 可计算delete device的操作耗时。 !image-2022-05-13-16-26-32-480.png! 100sg，500dev，20万序列 per dev。 delete脚本见附件。 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3177) [ concurrent delete dev ] NPE
刘珍 created IOTDB-3177: - Summary: [ concurrent delete dev ] NPE Key: IOTDB-3177 URL: https://issues.apache.org/jira/browse/IOTDB-3177 Project: Apache IoTDB Issue Type: Bug Components: Core/Schema Manager Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: ZhaoXin Attachments: del_dev_test.sh master_0510_19872aa 问题描述： 10用户并发delete dev.** ，有如下问题： 2022-05-12 14:20:21,337 [pool-117-IoTDB-RPC-Client-71] ERROR o.a.i.d.u.ErrorHandlingUtils:59 - Status code: INTERNAL_SERVER_ERROR(500), operation: executeNonQueryPlan failed java.lang.NullPointerException: null at org.apache.iotdb.db.metadata.mtree.store.CachedMTreeStore$CachedMNodeIterator.readNext(CachedMTreeStore.java:535) at org.apache.iotdb.db.metadata.mtree.store.CachedMTreeStore$CachedMNodeIterator.hasNext(CachedMTreeStore.java:507) at org.apache.iotdb.db.metadata.mtree.MTreeBelowSGCachedImpl.deleteTimeseriesAndReturnEmptyStorageGroup(MTreeBelowSGCachedImpl.java:394) at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionSchemaFileImpl.deleteOneTimeseriesUpdateStatisticsAndDropTrigger(SchemaRegionSchemaFileImpl.java:751) at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionSchemaFileImpl.deleteSingleTimeseriesInternal(SchemaRegionSchemaFileImpl.java:728) at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionSchemaFileImpl.deleteTimeseries(SchemaRegionSchemaFileImpl.java:704) at org.apache.iotdb.db.metadata.LocalSchemaProcessor.deleteTimeseries(LocalSchemaProcessor.java:364) at org.apache.iotdb.db.qp.executor.PlanExecutor.deleteTimeSeries(PlanExecutor.java:2067) at org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:334) at org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:53) at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:2066) at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryStatement(TSServiceImpl.java:1114) at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeUpdateStatement(TSServiceImpl.java:1107) at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeStatement(TSServiceImpl.java:622) at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeStatement.getResult(TSIService.java:2853) at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$executeStatement.getResult(TSIService.java:2833) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2022-05-12 14:20:21,694 [pool-117-IoTDB-RPC-Client-63] ERROR o.a.i.d.u.ErrorHandlingUtils:59 - Status code: INTERNAL_SERVER_ERROR(500), operation: executeNonQueryPlan failed java.lang.NullPointerException: null 操作流程 1. 192.168.10.68 使用数据备份启动数据库：/data/liuzhen_test/master_0510_19872aa/data_flush_stop_0511 复制这个备份，不要直接使用它。 2. 启动成功后，执行并发delete dev.**脚本 见附件 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3180) [Benchmark] fix the register of schema module in timescaledb and add experiment.
张洪胤 created IOTDB-3180: -- Summary: [Benchmark] fix the register of schema module in timescaledb and add experiment. Key: IOTDB-3180 URL: https://issues.apache.org/jira/browse/IOTDB-3180 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 fix the register of schema module in timescaledb and add experiment. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3197) [ Metadata recovery performance ] Different concurrent users create the same metadata, but the metadata recovery time varies greatly
刘珍 created IOTDB-3197:
-
Summary: [ Metadata recovery performance ] Different concurrent
users create the same metadata, but the metadata recovery time varies greatly
Key: IOTDB-3197
URL: https://issues.apache.org/jira/browse/IOTDB-3197
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Schema Manager
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: ZhaoXin
Attachments: config.properties.10client, config.properties.1client,
config.properties.20client, image-2022-05-16-13-20-13-848.png,
image-2022-05-16-13-20-31-354.png
master_0513_465b0cf
创建的元数据信息：
100sg，500dev，20w序列per dev , 共1亿序列
1用户/10用户/20用户并发创建元数据耗时
3127.65秒/675.27秒/520.38秒
但，这3份元数据恢复耗时：
332秒/{color:#DE350B}*1910秒/2364秒*{color}
10用户，20用户并发创建的元数据，在恢复的时候耗时长。
1用户恢复过程中监控cpu和io
!image-2022-05-16-13-20-13-848.png!
20用户恢复过程中健康cpu和io
!image-2022-05-16-13-20-31-354.png!
测试环境：
1. *192.168.10.6872C256G 1.2T HDD *9*
MAX_HEAP_SIZE="192G"
MAX_DIRECT_MEMORY_SIZE="32G"
schema_engine_mode=Schema_File
磁盘
2. benchmark生成数据的配置文件
见附件
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3198) [Metric] Move metric-related parameter from iotdb-engine to iotdb-metric.
张洪胤 created IOTDB-3198: -- Summary: [Metric] Move metric-related parameter from iotdb-engine to iotdb-metric. Key: IOTDB-3198 URL: https://issues.apache.org/jira/browse/IOTDB-3198 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 1. Remove enable_performance_stat parameter in iotdb-engine. 2. Add enablePerformanceStat in iotdb-metric. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3199) [Metric] Move metric-related parameter from iotdb-engine to iotdb-metric.
张洪胤 created IOTDB-3199: -- Summary: [Metric] Move metric-related parameter from iotdb-engine to iotdb-metric. Key: IOTDB-3199 URL: https://issues.apache.org/jira/browse/IOTDB-3199 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 1. Remove enable_performance_stat parameter in iotdb-engine. 2. Add enablePerformanceStat in iotdb-metric. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3045) [ schema recovery ] The query result contains some timeseries that have been deleted
[
https://issues.apache.org/jira/browse/IOTDB-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537439#comment-17537439
]
刘珍 commented on IOTDB-3045:
---
master_0516_7942b02，已解决。
> [ schema recovery ] The query result contains some timeseries that have been
> deleted
>
>
> Key: IOTDB-3045
> URL: https://issues.apache.org/jira/browse/IOTDB-3045
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Haiming Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.0-SNAPSHOT, 0.13.1
>
> Attachments: config.properties, d1.log
>
>
> master_0428_ca4f3cf
> 问题描述：
> 50 sg，100 dev，20 sensors/dev, 共2000万序列。
> 100个设备名称记录在文件中，
> 遍历文件，10个用户并发 delete ts，1个设备上delete 1条序列。
> 重复10次，也就是这100个dev，每个dev共delete 10个ts。
> delete 后，执行count timeseries dev.* ；结果符合预期。
> 备份数据。
> {color:#DE350B}重启iotdb，恢复元数据后，再查询 100个dev的序列数，有14个设备的序列数都是20万，
> 期望是10，delete的ts，又查询到了。{color}
> 详细测试流程：
> 1. 私有云 172.20.70.2(iotdb)/16(bm) 8C32G
> iotdb参数
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> schema_engine_mode=Schema_File
> 2. benchmark 配置见附件
> 3. 获取dev 名称，记录到文件dev_name.txt
> cat get_dev_name.sh
> #!/bin/bash
> ./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
> dev_name.txt
> 4. 并发delete
> cat del_1_ts.sh
> #!/bin/bash
> function delete_ts()
> {
>del_ts=$1
>dev_name=$2
>exp_count=$3
>count_ts ${exp_count} ${dev_name}
>./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
>let exp_count--
>count_ts ${exp_count} ${dev_name}
> }
> function count_ts()
> {
> exp_c_ts=$1
> dev_name=$2
> cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
> "4,4p"|awk -F '|' '{print $2}'`
> if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
> echo "ok"
> else
> echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
> fi
> }
> function execute()
> {
> in_file=$1
> i=$2
> exp_count=$3
> n=0
> cat ${in_file} | while read line
> do
>del_ts="${line}.s_${i}"
>dev_name="${line}.*";
>delete_ts ${del_ts} ${dev_name} ${exp_count} &
>let i++
>let n++
>b=$(( $n % 10 ))
>if [ "$b" -eq "0" ];then
> wait
>fi
> done
> wait
> }
> # 循环几轮
> count_ts_base=20
> for loop in {1..10}
> do
> execute "dev_name.txt" ${loop} ${count_ts_base}
> let count_ts_base--
> done
> 并发delete脚本的日志文件见附件。
> 执行count_ts.sh，查看每个dev的count timeseries，是否都是10。
> 有14个dev的结果是20，不符合预期结果。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (IOTDB-3004) [ metadata ] Log cannot fit into buffer, please increase mlog_buffer_size
[ https://issues.apache.org/jira/browse/IOTDB-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537858#comment-17537858 ] 刘珍 commented on IOTDB-3004: --- 使用模板，也有此问题： master_0516_7942b02，192.168.10.68机器 72C256G iotdb配置： MAX_HEAP_SIZE="192G" MAX_DIRECT_MEMORY_SIZE="32G" schema_engine_mode=Schema_File benchmark配置 TEMPLATE=true GROUP_NUMBER=100 DEVICE_NUMBER=500 SENSOR_NUMBER=20 CLIENT_NUMBER=10 DEVICE_NAME_PREFIX=test.schema_t.d_ !screenshot-1.png! 2022-05-16 18:12:04,918 [main] WARN o.a.i.c.s.StartupChecks:38 - iotdb.jmx.port missing from iotdb-env.sh(Unix or OS X, if you use Windows, check conf/iotdb-env.bat) 2022-05-16 19:35:02,606 [pool-16-IoTDB-RPC-Client-18] WARN o.a.i.d.u.ErrorHandlingUtils:153 - Exception occurred while processing non-query. org.apache.iotdb.db.exception.query.QueryProcessException: org.apache.iotdb.commons.exception.MetadataException: java.io.IOException: Log cannot fit into buffer, please increase mlog_buffer_size at org.apache.iotdb.db.qp.executor.PlanExecutor.createTemplate(PlanExecutor.java:474) at org.apache.iotdb.db.qp.executor.PlanExecutor.processNonQuery(PlanExecutor.java:405) at org.apache.iotdb.db.service.basic.StandaloneServiceProvider.executeNonQuery(StandaloneServiceProvider.java:53) at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.executeNonQueryPlan(TSServiceImpl.java:2061) at org.apache.iotdb.db.service.thrift.impl.TSServiceImpl.createSchemaTemplate(TSServiceImpl.java:1910) at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$createSchemaTemplate.getResult(TSIService.java:3754) at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$createSchemaTemplate.getResult(TSIService.java:3734) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.iotdb.commons.exception.MetadataException: java.io.IOException: Log cannot fit into buffer, please increase mlog_buffer_size at org.apache.iotdb.db.metadata.template.TemplateManager.createSchemaTemplate(TemplateManager.java:180) at org.apache.iotd > [ metadata ] Log cannot fit into buffer, please increase mlog_buffer_size > - > > Key: IOTDB-3004 > URL: https://issues.apache.org/jira/browse/IOTDB-3004 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Schema Manager >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Yukun Zhou >Priority: Major > Attachments: config.properties.20wperdev, > image-2022-04-25-18-18-16-223.png, iotdb-engine.properties, iotdb-env.sh, > screenshot-1.png > > > master_0425_907c06e > 问题描述： > 创建1000万的对齐序列（每个设备下有20万对齐序列），报错： > org.apache.iotdb.db.exception.metadata.MetadataException: > java.io.IOException: Log cannot fit into buffer, please increase > mlog_buffer_size > !image-2022-04-25-18-18-16-223.png! > 配置文件见附件（fit-3/4） -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3202) [heart beat] add some monitor metrics into heart beat request.
张洪胤 created IOTDB-3202: -- Summary: [heart beat] add some monitor metrics into heart beat request. Key: IOTDB-3202 URL: https://issues.apache.org/jira/browse/IOTDB-3202 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 add some monitor metrics into heart beat request. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3209) [ ConfigNode ] ConfigNode failed to start, restarting again is considered a restart
刘珍 created IOTDB-3209: - Summary: [ ConfigNode ] ConfigNode failed to start, restarting again is considered a restart Key: IOTDB-3209 URL: https://issues.apache.org/jira/browse/IOTDB-3209 Project: Apache IoTDB Issue Type: Bug Components: Core/Cluster Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Yongzao Dan Attachments: image-2022-05-17-11-17-39-917.png, image-2022-05-17-11-18-00-757.png, image-2022-05-17-11-19-03-968.png, image-2022-05-17-11-22-09-195.png master_0516_7942b02 iotdb-confignode.properties配置文件中的target_confignode参数配置的不正确，如： target_confignode=192.168.130.3:22277,192.168.130.4:22277 start-confignode.sh失败，却生成了 ./confignode/data/system/confignode-system.properties文件，里面记录的confignode_list=是空： !image-2022-05-17-11-19-03-968.png! iotdb-confignode.properties配置正确的target_confignode，再次启动confignode，失败： !image-2022-05-17-11-17-39-917.png! 根据异常信息，去代码查看，发现走了restart ： !image-2022-05-17-11-22-09-195.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3045) [ schema recovery ] The query result contains some timeseries that have been deleted
[
https://issues.apache.org/jira/browse/IOTDB-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537954#comment-17537954
]
刘珍 commented on IOTDB-3045:
---
0.13 db_0514_fe68d84，已解决。
> [ schema recovery ] The query result contains some timeseries that have been
> deleted
>
>
> Key: IOTDB-3045
> URL: https://issues.apache.org/jira/browse/IOTDB-3045
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Schema Manager
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Haiming Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.0-SNAPSHOT, 0.13.1
>
> Attachments: config.properties, d1.log
>
>
> master_0428_ca4f3cf
> 问题描述：
> 50 sg，100 dev，20 sensors/dev, 共2000万序列。
> 100个设备名称记录在文件中，
> 遍历文件，10个用户并发 delete ts，1个设备上delete 1条序列。
> 重复10次，也就是这100个dev，每个dev共delete 10个ts。
> delete 后，执行count timeseries dev.* ；结果符合预期。
> 备份数据。
> {color:#DE350B}重启iotdb，恢复元数据后，再查询 100个dev的序列数，有14个设备的序列数都是20万，
> 期望是10，delete的ts，又查询到了。{color}
> 详细测试流程：
> 1. 私有云 172.20.70.2(iotdb)/16(bm) 8C32G
> iotdb参数
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> schema_engine_mode=Schema_File
> 2. benchmark 配置见附件
> 3. 获取dev 名称，记录到文件dev_name.txt
> cat get_dev_name.sh
> #!/bin/bash
> ./sbin/start-cli.sh -e "show devices;" |grep root|awk -F '|' '{print $2}' >
> dev_name.txt
> 4. 并发delete
> cat del_1_ts.sh
> #!/bin/bash
> function delete_ts()
> {
>del_ts=$1
>dev_name=$2
>exp_count=$3
>count_ts ${exp_count} ${dev_name}
>./sbin/start-cli.sh -e "delete timeseries ${del_ts};"
>let exp_count--
>count_ts ${exp_count} ${dev_name}
> }
> function count_ts()
> {
> exp_c_ts=$1
> dev_name=$2
> cur_c_ts=`./sbin/start-cli.sh -e "count timeseries ${dev_name};"|sed -n
> "4,4p"|awk -F '|' '{print $2}'`
> if [ "${cur_c_ts}" -eq "${exp_c_ts}" ];then
> echo "ok"
> else
> echo "fail. ${dev_name} ${exp_c_ts} ${cur_c_ts}" >>./fail.log
> fi
> }
> function execute()
> {
> in_file=$1
> i=$2
> exp_count=$3
> n=0
> cat ${in_file} | while read line
> do
>del_ts="${line}.s_${i}"
>dev_name="${line}.*";
>delete_ts ${del_ts} ${dev_name} ${exp_count} &
>let i++
>let n++
>b=$(( $n % 10 ))
>if [ "$b" -eq "0" ];then
> wait
>fi
> done
> wait
> }
> # 循环几轮
> count_ts_base=20
> for loop in {1..10}
> do
> execute "dev_name.txt" ${loop} ${count_ts_base}
> let count_ts_base--
> done
> 并发delete脚本的日志文件见附件。
> 执行count_ts.sh，查看每个dev的count timeseries，是否都是10。
> 有14个dev的结果是20，不符合预期结果。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3214) [Metric] refactor predefined metric part.
张洪胤 created IOTDB-3214: -- Summary: [Metric] refactor predefined metric part. Key: IOTDB-3214 URL: https://issues.apache.org/jira/browse/IOTDB-3214 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 Refactor predefined metric part: # Add interface IMetricSet # Move file, process and system metric into predefined metrics. # Remove the difference between dropwizard and micrometer when use jvm predefined metric. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Reopened] (IOTDB-2900) [Benchmark] Improve schema module.
[ https://issues.apache.org/jira/browse/IOTDB-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张洪胤 reopened IOTDB-2900: > [Benchmark] Improve schema module. > -- > > Key: IOTDB-2900 > URL: https://issues.apache.org/jira/browse/IOTDB-2900 > Project: Apache IoTDB > Issue Type: Improvement > Components: Benchmark >Reporter: 张洪胤 >Assignee: 张洪胤 >Priority: Major > Original Estimate: 120h > Remaining Estimate: 120h > > Improve schema module to support more situation. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3220) Template recovery is slow
刘珍 created IOTDB-3220: - Summary: Template recovery is slow Key: IOTDB-3220 URL: https://issues.apache.org/jira/browse/IOTDB-3220 Project: Apache IoTDB Issue Type: Improvement Components: Core/Schema Manager Reporter: 刘珍 Assignee: ZhaoXin Attachments: config.properties, image-2022-05-17-17-44-54-106.png master_0516_7942b02 template包含20万序列，data/system/schema/template_log.bin 大小7.6MB，恢复慢，耗时1小时20分钟，需优化。 !image-2022-05-17-17-44-54-106.png! 测试流程: 1. 192.168.10.68 机器72C256G MAX_HEAP_SIZE="192G" MAX_DIRECT_MEMORY_SIZE="32G" mlog_buffer_size=10485760 schema_engine_mode=Schema_File 2. benchmark配置见附件，生成数据，再启动恢复，可复现template恢复慢。 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3225) Solutions for corrupted files due to compaction before 0.12.4
周沛辰 created IOTDB-3225: -- Summary: Solutions for corrupted files due to compaction before 0.12.4 Key: IOTDB-3225 URL: https://issues.apache.org/jira/browse/IOTDB-3225 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3233) [metric] add statistics of device and storage group in metric framework.
张洪胤 created IOTDB-3233: -- Summary: [metric] add statistics of device and storage group in metric framework. Key: IOTDB-3233 URL: https://issues.apache.org/jira/browse/IOTDB-3233 Project: Apache IoTDB Issue Type: New Feature Reporter: 张洪胤 Assignee: 张洪胤 Add statistic of device and storage group in metric framework. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Reopened] (IOTDB-2978) [compaction error log ] Log level and "null" error message handling
[ https://issues.apache.org/jira/browse/IOTDB-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 刘珍 reopened IOTDB-2978: --- master_0519_81b9117 bm写入过程中执行./sbin/stop-server.sh 2022-05-19 16:22:53,647 [pool-7-IoTDB-Compaction-7] ERROR o.a.i.t.r.r.LocalTsFileInput:92 - Error happened while reading /data/iotdb_data/data/sequence/root.test.g_4/0/0/1652947428948-27-0-0.tsfile from position 2768673 2022-05-19 16:22:53,652 [pool-7-IoTDB-Compaction-7] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:203 - root.test.g_4-0 [Compaction] Meet errors in inner space compaction. 2022-05-19 16:22:53,683 [pool-7-IoTDB-Compaction-6] ERROR o.a.i.t.r.r.LocalTsFileInput:92 - Error happened while reading /data/iotdb_data/data/sequence/root.test.g_3/0/0/1652946897949-7-0-0.tsfile from position 4735428 2022-05-19 16:22:53,683 [pool-7-IoTDB-Compaction-6] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:203 - root.test.g_3-0 [Compaction] Meet errors in inner space compaction. 2022-05-19 16:22:53,702 [pool-7-IoTDB-Compaction-7] ERROR o.a.i.d.e.c.t.AbstractCompactionTask:80 - root.test.g_4-0 [Compaction] Running compaction task failed java.nio.channels.ClosedByInterruptException: null at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:746) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) at org.apache.iotdb.tsfile.read.reader.LocalTsFileInput.read(LocalTsFileInput.java:90) at org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readAsPossible(ReadWriteIOUtils.java:771) at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readData(TsFileSequenceReader.java:1194) at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readChunk(TsFileSequenceReader.java:1065) at org.apache.iotdb.tsfile.read.TsFileSequenceReader.readMemChunk(TsFileSequenceReader.java:1078) at org.apache.iotdb.tsfile.read.TsFileAlignedSeriesReaderIterator.nextReader(TsFileAlignedSeriesReaderIterator.java:76) at org.apache.iotdb.db.engine.compaction.inner.utils.AlignedSeriesCompactionExecutor.execute(AlignedSeriesCompactionExecutor.java:122) at org.apache.iotdb.db.engine.compaction.performer.impl.ReadChunkCompactionPerformer.compactAlignedSeries(ReadChunkCompactionPerformer.java:115) at org.apache.iotdb.db.engine.compaction.performer.impl.ReadChunkCompactionPerformer.perform(ReadChunkCompactionPerformer.java:77) at org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask.doCompaction(InnerSpaceCompactionTask.java:129) at org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.call(AbstractCompactionTask.java:74) at org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.call(AbstractCompactionTask.java:40) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) > [compaction error log ] Log level and "null" error message handling > - > > Key: IOTDB-2978 > URL: https://issues.apache.org/jira/browse/IOTDB-2978 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Compaction >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: 周沛辰 >Priority: Minor > Labels: pull-request-available > Fix For: 0.14.0 > > Attachments: image-2022-04-22-10-24-41-780.png > > > master_0420_6bd0f2c > stop-server.sh compaction error log： > 问题1 ：status: abort 日志级别需处理为warn 或info > 问题2：ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_44-0 > [Compaction] Throwable is caught during execution of SizeTieredCompaction, > null > 需处理 > 问题3：ERROR o.a.i.t.r.r.LocalTsFileInput:92 - Error happened while reading > /data/iotdb_data/data/sequence/root.test.g_44/0/0/1650443726681-274-0-0.tsfile > from position 3490553 > 需处理 > !image-2022-04-22-10-24-41-780.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3241) [Config] fix the place of dir in common config.
张洪胤 created IOTDB-3241: -- Summary: [Config] fix the place of dir in common config. Key: IOTDB-3241 URL: https://issues.apache.org/jira/browse/IOTDB-3241 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 fix the user folder and role folder place in common config. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3245) [Benchmark] fix the release of source when exceptions happen.
张洪胤 created IOTDB-3245: -- Summary: [Benchmark] fix the release of source when exceptions happen. Key: IOTDB-3245 URL: https://issues.apache.org/jira/browse/IOTDB-3245 Project: Apache IoTDB Issue Type: Bug Components: Benchmark Reporter: 张洪胤 Assignee: 张洪胤 When iotdb is done, iotdb-benchmark can't stop because the thread pool in record haven't been closed. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3246) TsFile Validation Tool
周沛辰 created IOTDB-3246: -- Summary: TsFile Validation Tool Key: IOTDB-3246 URL: https://issues.apache.org/jira/browse/IOTDB-3246 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 This tool can be used to check the correctness of tsfile and point out errors in specific timeseries or devices. The types of errors include the following: * Device overlap between files * Timeseries overlap between files * Timeseries overlap between chunks * Timeseries overlap between pages * Timeseries overlap within one page -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3247) [wal recovery] Aligned sensors, query lost data
刘珍 created IOTDB-3247:
-
Summary: [wal recovery] Aligned sensors, query lost data
Key: IOTDB-3247
URL: https://issues.apache.org/jira/browse/IOTDB-3247
Project: Apache IoTDB
Issue Type: Bug
Components: Core/WAL
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Haiming Zhu
Attachments: image-2022-05-20-16-09-01-848.png
master_0519_81b9117
问题描述：
100sg，500dev，20万序列/dev，共1亿对齐序列，每个序列写入10个点。
每个device，delete 51个序列，重启iotdb，wal恢复有2个问题：
问题1：未被delete的部分序列，{color:#DE350B}*查询少数据*{color}（值小于10）
问题2：恢复过程中有NPE
2022-05-20 14:16:09,213 [pool-15-IoTDB-WAL-Recover-2] WARN
o.a.i.d.w.r.f.UnsealedTsFileRecoverPerformer:208 - meet error when redo wal of
/data/liuzhen_test/master_0519_81b9117/datanode/./sbin/../data/data/sequence/root.test.g_99/0/0/1652977295224-2-0-0.tsfile
org.apache.iotdb.db.exception.WriteProcessException:
java.lang.NullPointerException
at
org.apache.iotdb.db.engine.memtable.AbstractMemTable.insertAlignedTablet(AbstractMemTable.java:394)
at
org.apache.iotdb.db.wal.recover.file.TsFilePlanRedoer.redoInsert(TsFilePlanRedoer.java:128)
at
org.apache.iotdb.db.wal.recover.file.UnsealedTsFileRecoverPerformer.redoLog(UnsealedTsFileRecoverPerformer.java:191)
at
org.apache.iotdb.db.wal.recover.WALNodeRecoverTask.recoverTsFiles(WALNodeRecoverTask.java:137)
at
org.apache.iotdb.db.wal.recover.WALNodeRecoverTask.run(WALNodeRecoverTask.java:63)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
at
org.apache.iotdb.db.utils.datastructure.AlignedTVList.arrayCopy(AlignedTVList.java:808)
at
org.apache.iotdb.db.utils.datastructure.AlignedTVList.putAlignedValues(AlignedTVList.java:736)
at
org.apache.iotdb.db.engine.memtable.AlignedWritableMemChunk.putAlignedValues(AlignedWritableMemChunk.java:152)
at
org.apache.iotdb.db.engine.memtable.AlignedWritableMemChunk.writeAlignedValues(AlignedWritableMemChunk.java:182)
at
org.apache.iotdb.db.engine.memtable.AlignedWritableMemChunkGroup.writeValues(AlignedWritableMemChunkGroup.java:55)
at
org.apache.iotdb.db.engine.memtable.AbstractMemTable.writeAlignedTablet(AbstractMemTable.java:545)
at
org.apache.iotdb.db.engine.memtable.AbstractMemTable.insertAlignedTablet(AbstractMemTable.java:377)
... 9 common frames omitted
测试流程
1. 192.168.10.68 72C256G
iotdb路径：/data/liuzhen_test/master_0519_81b9117/datanode
iotdb配置（其余不改动）：
MAX_HEAP_SIZE="192G"
MAX_DIRECT_MEMORY_SIZE="32G"
mlog_buffer_size=10485760
schema_engine_mode=Schema_File
benchmark路径：/data/benchmark/weekly_shell/bm_0514_ee75a49
bm配置见附件。
2. 启动iotdb，运行benchmark
耗时大概3小时。
3. delete 序列前的数据验证
正确
count_ts_500dev.sh 每个设备20万序列
select_count_ts_500dev.sh 查询序列10个点数据。
4. 每个设备delete 51个序列
运行del_ts.sh
5. delete 序列后，停止iotdb前，再次验证数据的正确性
正确
count_ts_500dev.sh 每个设备199949序列
select_count_ts_500dev.sh 查询序列10个点数据。
6.停止iotdb
7. 备份数据，日志
8.重新启动iotdb，查看日志，有NPE
9. iotdb恢复成功，执行
select_count_ts_500dev.sh {color:#DE350B}*部分少数据的序列*{color}（只列举部分）
!image-2022-05-20-16-09-01-848.png!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3258) Support Count Nodes in node management
张洪胤 created IOTDB-3258: -- Summary: Support Count Nodes in node management Key: IOTDB-3258 URL: https://issues.apache.org/jira/browse/IOTDB-3258 Project: Apache IoTDB Issue Type: New Feature Reporter: 张洪胤 Assignee: 张洪胤 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-3168) The path with * could be executed successfully when inserting
[ https://issues.apache.org/jira/browse/IOTDB-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 武思兴 reassigned IOTDB-3168: -- Assignee: 武思兴 > The path with * could be executed successfully when inserting > - > > Key: IOTDB-3168 > URL: https://issues.apache.org/jira/browse/IOTDB-3168 > Project: Apache IoTDB > Issue Type: Bug >Reporter: Wei Liu >Assignee: 武思兴 >Priority: Major > Fix For: 0.14.0, 0.13.1 > > Attachments: screenshot-1.png > > > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3261) [ 3C3D ] Internal error processing getOrCreateSchemaPartition NPE
刘珍 created IOTDB-3261:
-
Summary: [ 3C3D ] Internal error processing
getOrCreateSchemaPartition NPE
Key: IOTDB-3261
URL: https://issues.apache.org/jira/browse/IOTDB-3261
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Schema Manager
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Yongzao Dan
master_0523_51e9703
3C3D
并发创建schema，有如下异常（sg创建成功，show devices报错）
2022-05-23 11:46:29,291 [pool-1-IoTDB-ConfigNodeRPC-Client-53] ERROR
o.a.t.ProcessFunction:47 - Internal error processing getOrCreateSchemaPartition
java.lang.NullPointerException: null
at
org.apache.iotdb.confignode.manager.load.balancer.allocator.CopySetRegionAllocator.clear(CopySetRegionAllocator.java:165)
at
org.apache.iotdb.confignode.manager.load.balancer.allocator.CopySetRegionAllocator.allocateRegion(CopySetRegionAllocator.java:77)
at
org.apache.iotdb.confignode.manager.load.balancer.RegionBalancer.genRegionsAllocationPlan(RegionBalancer.java:87)
at
org.apache.iotdb.confignode.manager.load.LoadManager.initializeRegions(LoadManager.java:92)
at
org.apache.iotdb.confignode.manager.PartitionManager.checkAndAllocateRegionsIfNecessary(PartitionManager.java:295)
at
org.apache.iotdb.confignode.manager.PartitionManager.getOrCreateSchemaPartition(PartitionManager.java:102)
at
org.apache.iotdb.confignode.manager.ConfigManager.getOrCreateSchemaPartition(ConfigManager.java:360)
at
org.apache.iotdb.confignode.service.thrift.ConfigNodeRPCServiceProcessor.getOrCreateSchemaPartition(ConfigNodeRPCServiceProcessor.java:259)
at
org.apache.iotdb.confignode.rpc.thrift.ConfigIService$Processor$getOrCreateSchemaPartition.getResult(ConfigIService.java:1715)
at
org.apache.iotdb.confignode.rpc.thrift.ConfigIService$Processor$getOrCreateSchemaPartition.getResult(ConfigIService.java:1695)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
IoTDB> show devices;
{color:#DE350B}*Msg: 500: [INTERNAL_SERVER_ERROR(500)] Exception occurred:
"show devices". executeStatement failed. null*{color}
show storage group; 结果正确。
1. 机器信息
192.168.130.3/4/5
benchmark配置文件见附件。
confignode log error见附件。
datanode log error见附件。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3262) [ cluster metadata ] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 201 is too large...
刘珍 created IOTDB-3262: - Summary: [ cluster metadata ] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 201 is too large... Key: IOTDB-3262 URL: https://issues.apache.org/jira/browse/IOTDB-3262 Project: Apache IoTDB Issue Type: Bug Components: Core/Schema Manager Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Yukun Zhou Attachments: image-2022-05-23-14-40-36-059.png, image-2022-05-23-14-42-29-889.png master_0523_51e9703 3C3D 创建30sg，300dev，1k ts/dev，共30万序列。 datanode内存：-Xmx7968M 创建元数据失败： confignode 没有error，有WARN： 2022-05-23 13:53:31,618 [192.168.130.3-22278@group-->192.168.130.5-22278-GrpcLogAppender-LogAppenderDaemon] WARN o.a.r.g.s.GrpcLogAppender:177 - 192.168.130.3-22278@group-->192.168.130.5-22278-GrpcLogAppender: Wait interrupted by java.lang.InterruptedException 2022-05-23 13:58:21,715 [192.168.130.3-22278@group--StateMachineUpdater] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 201 is too large... 2022-05-23 13:58:21,718 [192.168.130.3-22278@group--StateMachineUpdater] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 432 is too large... 2022-05-23 13:58:21,743 [192.168.130.3-22278@group--StateMachineUpdater] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 663 is too large... !image-2022-05-23-14-42-29-889.png! datanode 报错： 2022-05-23 13:58:25,843 [pool-18-IoTDB-MPPCoordinatorWrite-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal error processing sendFragmentInstance, LocalizedMessage Internal error processing sendFragmentInstance, org.apache.thrift.TApplicationException: Internal error processing sendFragmentInstance at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.CGLIB$receiveBase$51() at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446$$FastClassByCGLIB$$76030a70.invoke() at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55) at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.receiveBase() at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_sendFragmentInstance(InternalService.java:170) at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.CGLIB$recv_sendFragmentInstance$12() at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446$$FastClassByCGLIB$$76030a70.invoke() at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55) at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.recv_sendFragmentInstance() at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.sendFragmentInstance(InternalService.java:157) at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.CGLIB$sendFragmentInstance$39() at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446$$FastClassByCGLIB$$76030a70.invoke() at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55) at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.sendFragmentInstance() at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:157) at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:138) at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchWrite$1(FragmentInstanceDispatcherImpl.java:110) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) !image-2022-05-23-14-40-36-059.png! 复现流程： 1. 机器信息 192.168.130.3/4/5 16C32G 2. benchmark 配置见附件 使用的bm版本：/home/benchmark/bm_20220322 3. iotdb配置文件见附件 -- This message was sent by Atl
[jira] [Created] (IOTDB-3270) [last cache] fix last cache.
张洪胤 created IOTDB-3270: -- Summary: [last cache] fix last cache. Key: IOTDB-3270 URL: https://issues.apache.org/jira/browse/IOTDB-3270 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 Make lastCacheEnable in use. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3272) [ error message ] Borrow client error when the target DataNode is down
刘珍 created IOTDB-3272:
-
Summary: [ error message ] Borrow client error when the target
DataNode is down
Key: IOTDB-3272
URL: https://issues.apache.org/jira/browse/IOTDB-3272
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Quan Siyi
Attachments: image-2022-05-24-10-34-58-400.png
master_0523_51e9703
data_replication_factor=1
3confignode，3datanode。
停止node1的datanode，cli在node2 查询node1上的数据，{color:#DE350B}*报错信息不明确*{color}：
!image-2022-05-24-10-34-58-400.png!
error code: TSStatus(code:411,
message:org.apache.iotdb.db.exception.mpp.FragmentInstanceDispatchException:
java.io.IOException: {color:#DE350B}*Borrow client from pool for node
TEndPoint(ip:192.168.130.3, port:9003) failed.)*{color}
2022-05-24 10:27:25,667 [pool-17-IoTDB-MPPCoordinator-3] ERROR
o.a.i.d.m.p.s.SimpleQueryTerminator:78 - can't connect to node
TEndPoint(ip:192.168.130.3, port:9003)
java.io.IOException: {color:#DE350B}*Borrow client from pool for node
TEndPoint(ip:192.168.130.3, port:9003) failed.*{color}
at
org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:61)
at
org.apache.iotdb.db.mpp.plan.scheduler.SimpleQueryTerminator.lambda$terminate$0(SimpleQueryTerminator.java:74)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: net.sf.cglib.core.CodeGenerationException:
org.apache.thrift.transport.TTransportException-->java.net.ConnectException:
Connection refused (Connection refused)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:235)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:220)
at net.sf.cglib.proxy.Enhancer.createUsingReflection(Enhancer.java:639)
at net.sf.cglib.proxy.Enhancer.firstInstance(Enhancer.java:538)
at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:231)
at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
at
org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.newErrorHandler(SyncThriftClientWithErrorHandler.java:48)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:127)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$Factory.makeObject(SyncDataNodeInternalServiceClient.java:105)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:780)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:439)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
at
org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:50)
... 5 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:243)
at
org.apache.iotdb.rpc.TElasticFramedTransport.open(TElasticFramedTransport.java:91)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient.(SyncDataNodeInternalServiceClient.java:63)
at
org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$33430b8f.()
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:228)
... 18 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394)
at java.net.Socket.connect(Socket.java:606)
at org.apache.thrift.transport.TSocket.open(TSocket.java:238)
... 26 common frames omitted
[jira] [Created] (IOTDB-3276) [ datanode restart ] tsfile is all deleted （ [CrossSpaceCompaction] Delete TsFile）
刘珍 created IOTDB-3276: - Summary: [ datanode restart ] tsfile is all deleted （ [CrossSpaceCompaction] Delete TsFile） Key: IOTDB-3276 URL: https://issues.apache.org/jira/browse/IOTDB-3276 Project: Apache IoTDB Issue Type: Bug Components: Core/Compaction Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: 周沛辰 Attachments: image-2022-05-24-13-19-18-400.png master_0523_51e9703 问题描述： 1. 单副本data_replication_factor=1 3confignode，3datanode 2.bm连node1，写入数据（如root.sg1.dev1），数据在node1。 kill node1的datanode进程。 3. bm连node2，写入数据（root.sg1.dev2）,数据在node2。 4. 启动node1的datanode，node1上的tsfile被完全删除： 2022-05-24 12:36:20,574 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653363632954-1-0-0.tsfile. 2022-05-24 12:36:20,574 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653363736139-2-0-0.tsfile. 2022-05-24 12:36:20,574 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653363838825-3-0-0.tsfile. 2022-05-24 12:36:20,574 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653363941370-4-0-0.tsfile. 2022-05-24 12:36:20,574 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653364044379-5-0-0.tsfile. 2022-05-24 12:36:20,574 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653364148312-6-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653364251418-7-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653364354126-8-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653364456635-9-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/sequence/root.test.g_0/1/0/1653364559232-10-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/unsequence/root.test.g_0/1/0/1653366964624-11-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/unsequence/root.test.g_0/1/0/1653366965016-12-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/unsequence/root.test.g_0/1/0/1653366965401-13-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/unsequence/root.test.g_0/1/0/1653366965655-14-0-0.tsfile. 2022-05-24 12:36:20,575 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/unsequence/root.test.g_0/1/0/1653366965862-15-0-0.tsfile. 2022-05-24 12:36:20,576 [pool-7-IoTDB-Compaction-1] INFO o.a.i.d.e.c.c.CrossSpaceCompactionTask:264 - [CrossSpaceCompaction] Delete TsFile :/data3/cluster_test/master_0523_51e9703/datanode/./sbin/../data/data/unsequence
[jira] [Assigned] (IOTDB-3168) The path with * could be executed successfully when inserting
[ https://issues.apache.org/jira/browse/IOTDB-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 武思兴 reassigned IOTDB-3168: -- Assignee: 武思兴 (was: liaolanyu) > The path with * could be executed successfully when inserting > - > > Key: IOTDB-3168 > URL: https://issues.apache.org/jira/browse/IOTDB-3168 > Project: Apache IoTDB > Issue Type: Bug >Reporter: Wei Liu >Assignee: 武思兴 >Priority: Major > Fix For: 0.14.0, 0.13.1 > > Attachments: screenshot-1.png > > > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3278) [ cluster OOM ] java.lang.OutOfMemoryError: GC overhead limit exceeded
刘珍 created IOTDB-3278: - Summary: [ cluster OOM ] java.lang.OutOfMemoryError: GC overhead limit exceeded Key: IOTDB-3278 URL: https://issues.apache.org/jira/browse/IOTDB-3278 Project: Apache IoTDB Issue Type: Bug Components: mpp-cluster Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Haonan Hou Attachments: config.properties, image-2022-05-24-18-00-10-598.png, image-2022-05-24-18-03-10-418.png, image-2022-05-24-18-04-56-541.png, ip3_iotdb-confignode.properties, ip3_iotdb-engine.properties, ip4_iotdb-confignode.properties, ip4_iotdb-engine.properties, ip5_iotdb-confignode.properties, ip5_iotdb-engine.properties master_0523_51e9703 问题描述： 1. data_replication_factor=1 3confignode，3datanode bm （SESSION_BY_TABLET）：1用户insert 1dev，100sensor，BATCH_SIZE_PER_WRITE=1000 bm运行起来后，就收到： !image-2022-05-24-18-00-10-598.png! 但此时连接的datanode的日志中并没有这个异常（error，warn）。 过5，6分钟，连接的datanode的warn： !image-2022-05-24-18-03-10-418.png! 8分钟，连接的datanode的error： !image-2022-05-24-18-04-56-541.png! 复现流程： 1. 3台测试机器192.168.130.3/4/5 confignode，datanode的配置文件见附件。 2. bm配置文件见附件(config.properties连ip-3) bm在192.168.130.2 /home/benchmark/bm_20220322 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3225) Solutions for corrupted files due to compaction before 0.12.4
[ https://issues.apache.org/jira/browse/IOTDB-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17541797#comment-17541797 ] 周沛辰 commented on IOTDB-3225: This tool can only check correctness of device between tsfiles and has been replaced by TsFileValidationTool. > Solutions for corrupted files due to compaction before 0.12.4 > - > > Key: IOTDB-3225 > URL: https://issues.apache.org/jira/browse/IOTDB-3225 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: 周沛辰 >Assignee: 周沛辰 >Priority: Major > Labels: pull-request-available > Original Estimate: 16h > Remaining Estimate: 16h > > Create a repair tool to move corrupted seq files into unseq dir. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-3165) Implement data deletion in distributed system
[ https://issues.apache.org/jira/browse/IOTDB-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 周沛辰 reassigned IOTDB-3165: -- Assignee: Liuxuxin (was: 周沛辰) > Implement data deletion in distributed system > - > > Key: IOTDB-3165 > URL: https://issues.apache.org/jira/browse/IOTDB-3165 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > > Implement data deletion in distributed system. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3282) RewriteBadFileTool for corrupted files due to compaction before 0.12.4
周沛辰 created IOTDB-3282: -- Summary: RewriteBadFileTool for corrupted files due to compaction before 0.12.4 Key: IOTDB-3282 URL: https://issues.apache.org/jira/browse/IOTDB-3282 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 Create a rewriteBadFileTool to move bad files and rewrite them into iotdb. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3285) createAlignedTimeseries failed java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
刘珍 created IOTDB-3285: - Summary: createAlignedTimeseries failed java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 Key: IOTDB-3285 URL: https://issues.apache.org/jira/browse/IOTDB-3285 Project: Apache IoTDB Issue Type: Bug Components: Core/Schema Manager Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Yukun Zhou Attachments: config.properties, image-2022-05-25-13-59-40-493.png master_0525_5f8c634 1confignode , 1datanode schema_replication_factor=1 data_replication_factor=1 注册对齐序列失败 2022-05-25 11:45:26,700 [pool-20-IoTDB-RPC-Client-2] WARN o.a.i.d.u.ErrorHandlingUtils:61 - Status code: EXECUTE_STATEMENT_ERROR(400), operation: createAlignedTimeseries failed java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList$SubList.rangeCheck(ArrayList.java:1225) at java.util.ArrayList$SubList.get(ArrayList.java:1042) at org.apache.iotdb.db.mpp.common.schematree.PathPatternTree.parseNodesToString(PathPatternTree.java:131) at org.apache.iotdb.db.mpp.common.schematree.PathPatternTree.findAllDevicePaths(PathPatternTree.java:112) at org.apache.iotdb.db.mpp.common.schematree.PathPatternTree.findAllDevicePaths(PathPatternTree.java:103) at org.apache.iotdb.db.mpp.plan.analyze.ClusterPartitionFetcher.getOrCreateSchemaPartition(ClusterPartitionFetcher.java:134) at org.apache.iotdb.db.mpp.plan.analyze.Analyzer$AnalyzeVisitor.visitCreateAlignedTimeseries(Analyzer.java:958) at org.apache.iotdb.db.mpp.plan.analyze.Analyzer$AnalyzeVisitor.visitCreateAlignedTimeseries(Analyzer.java:128) at org.apache.iotdb.db.mpp.plan.statement.metadata.CreateAlignedTimeSeriesStatement.accept(CreateAlignedTimeSeriesStatement.java:191) at org.apache.iotdb.db.mpp.plan.statement.StatementVisitor.process(StatementVisitor.java:62) at org.apache.iotdb.db.mpp.plan.analyze.Analyzer.analyze(Analyzer.java:120) at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.analyze(QueryExecution.java:182) at org.apache.iotdb.db.mpp.plan.execution.QueryExecution.(QueryExecution.java:121) at org.apache.iotdb.db.mpp.plan.Coordinator.createQueryExecution(Coordinator.java:107) at org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:127) at org.apache.iotdb.db.service.thrift.impl.DataNodeTSIServiceImpl.createAlignedTimeseries(DataNodeTSIServiceImpl.java:403) at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$createAlignedTimeseries.getResult(TSIService.java:3178) at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$createAlignedTimeseries.getResult(TSIService.java:3158) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) !image-2022-05-25-13-59-40-493.png! 复现流程 1. 192.168.10.68 72C256G benchmark 配置文件见附件 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3287) [TTL] add precheck of ttl in set sg and set ttl.
张洪胤 created IOTDB-3287: -- Summary: [TTL] add precheck of ttl in set sg and set ttl. Key: IOTDB-3287 URL: https://issues.apache.org/jira/browse/IOTDB-3287 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 Add precheck of ttl in set sg and set ttl: tll should greater than 0. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3288) [Doc] update auth doc
张洪胤 created IOTDB-3288: -- Summary: [Doc] update auth doc Key: IOTDB-3288 URL: https://issues.apache.org/jira/browse/IOTDB-3288 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3289) [ schema region ] The created schema region is greater than "maximum_schema_region_count "(default value =4)
刘珍 created IOTDB-3289:
-
Summary: [ schema region ] The created schema region is greater
than "maximum_schema_region_count "(default value =4)
Key: IOTDB-3289
URL: https://issues.apache.org/jira/browse/IOTDB-3289
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Schema Manager
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Yukun Zhou
Attachments: config.properties
master_0525_5f8c634
问题描述
maximum_schema_region_count = 4（默认值）
此测试场景出现每个{color:#DE350B}*sg下的schema region数为6*{color}
复现步骤
1. 192.168.10.68
schema_replication_factor=1
data_replication_factor=1
1confignode，1datanode
benchmark运行附件中的配置
问题1 ：创建的sg的schema region 为6 。
问题2：benchmark客户端收到异常（datanode/confignode没有这个异常）
2022-05-25 14:20:37,520 ERROR
cn.edu.tsinghua.iotdb.benchmark.iotdb013.IoTDB:323 - Register IoTDB schema
failed because
org.apache.iotdb.rpc.StatementExecutionException: 411:
java.util.concurrent.ExecutionException:
java.lang.UnsupportedOperationException: Constructor threw an exception for
org.apache.ratis.statemachine.impl.TransactionContextImpl
at org.apache.iotdb.rpc.RpcUtils.verifySuccess(RpcUtils.java:94)
at
org.apache.iotdb.session.SessionConnection.createMultiTimeseries(SessionConnection.java:300)
at
org.apache.iotdb.session.Session.createMultiTimeseries(Session.java:555)
at
cn.edu.tsinghua.iotdb.benchmark.iotdb013.IoTDB.registerTimeseriesBatch(IoTDB.java:308)
at
cn.edu.tsinghua.iotdb.benchmark.iotdb013.IoTDB.registerTimeseries(IoTDB.java:268)
at
cn.edu.tsinghua.iotdb.benchmark.iotdb013.IoTDB.registerSchema(IoTDB.java:156)
at
cn.edu.tsinghua.iotdb.benchmark.tsdb.DBWrapper.registerSchema(DBWrapper.java:509)
at
cn.edu.tsinghua.iotdb.benchmark.mode.BaseMode.registerSchema(BaseMode.java:111)
at
cn.edu.tsinghua.iotdb.benchmark.mode.TestWithDefaultPathMode.preCheck(TestWithDefaultPathMode.java:48)
at cn.edu.tsinghua.iotdb.benchmark.mode.BaseMode.run(BaseMode.java:62)
at cn.edu.tsinghua.iotdb.benchmark.App.main(App.java:61)
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3291) system/storage_groups ：Too many folders under each storage group
刘珍 created IOTDB-3291: - Summary: system/storage_groups ：Too many folders under each storage group Key: IOTDB-3291 URL: https://issues.apache.org/jira/browse/IOTDB-3291 Project: Apache IoTDB Issue Type: Bug Components: Core/Schema Manager Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Jialin Qiao Attachments: config.properties, image-2022-05-25-16-09-52-128.png master_0525_5f8c634 system/storage_groups 下每个存储组有30个文件夹，太多了。 !image-2022-05-25-16-09-52-128.png! 复现流程 1 . 192.168.10.68 72C256G schema_replication_factor=1 data_replication_factor=1 1confignode，1datanode benchmark运行附件中的配置。 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (IOTDB-3282) RewriteBadFileTool for corrupted files due to compaction before 0.12.4
[ https://issues.apache.org/jira/browse/IOTDB-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17542301#comment-17542301 ] 周沛辰 commented on IOTDB-3282: This tool will do the following two things: (1) remove the bad files validated by ValidationTool to the backUp dir. (2) rewrite them into iotdb by insertTablet interface. > RewriteBadFileTool for corrupted files due to compaction before 0.12.4 > -- > > Key: IOTDB-3282 > URL: https://issues.apache.org/jira/browse/IOTDB-3282 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: 周沛辰 >Assignee: 周沛辰 >Priority: Major > Attachments: rewriteBadFile-0.12.6-SNAPSHOT(2).zip > > Original Estimate: 32h > Remaining Estimate: 32h > > Create a rewriteBadFileTool to move bad files and rewrite them into iotdb. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3307) [Benchmark] Fix the start timestamp of last device in each client.
张洪胤 created IOTDB-3307: -- Summary: [Benchmark] Fix the start timestamp of last device in each client. Key: IOTDB-3307 URL: https://issues.apache.org/jira/browse/IOTDB-3307 Project: Apache IoTDB Issue Type: Bug Components: Benchmark Reporter: 张洪胤 Assignee: 张洪胤 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3315) [TTL] optimize not set of ttl
张洪胤 created IOTDB-3315: -- Summary: [TTL] optimize not set of ttl Key: IOTDB-3315 URL: https://issues.apache.org/jira/browse/IOTDB-3315 Project: Apache IoTDB Issue Type: Bug Reporter: 张洪胤 Assignee: 张洪胤 Make unset of ttl in cluster mode same as 0.13.0 version. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Reopened] (IOTDB-3040) [ TTL ] Expired tsfiles with a TTL of more than 10 hours are not deleted
[
https://issues.apache.org/jira/browse/IOTDB-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
刘珍 reopened IOTDB-3040:
---
master_0527_15fa098 TTL 又有新现象：
192.168.130.14机器
[cluster@i-g573lvjv logs]$ grep "by ttl " log_all.log |grep "root.test.g_44"
|tail -4
2022-05-30 10:43:51,944 [pool-12-IoTDB-TTL-Check-1] INFO
o.a.i.d.e.s.DataRegion:1677 - Removed a file
./sbin/../data/data/sequence/root.test.g_44/0/0/*{color:red}1653750006773-4771-1{color}*-0.tsfile
before Thu Aug 30 21:32:18 CST 2018 by ttl (118242693000ms)
去root.test.g_44下ll一下，可以看到4500到4801之间的300个tsfile已经被TTL删除，但是更为过期的1653704195143-2821-0-0.tsfile到1653743555186-4500-0-0.tsfile则没有被删除，产生了堆积。
-rw-rw-r-- 1 cluster cluster 1800 May 28 21:11
1653743440789-4495-0-0.tsfile.resource
-rw-rw-r-- 1 cluster cluster 14375438 May 28 21:11 1653743464051-4496-0-0.tsfile
-rw-rw-r-- 1 cluster cluster 1800 May 28 21:11
1653743464051-4496-0-0.tsfile.resource
-rw-rw-r-- 1 cluster cluster 14951671 May 28 21:12 1653743485800-4497-0-0.tsfile
-rw-rw-r-- 1 cluster cluster 1800 May 28 21:12
1653743485800-4497-0-0.tsfile.resource
-rw-rw-r-- 1 cluster cluster 14565793 May 28 21:12 1653743509895-4498-0-0.tsfile
-rw-rw-r-- 1 cluster cluster 1800 May 28 21:12
1653743509895-4498-0-0.tsfile.resource
-rw-rw-r-- 1 cluster cluster 14793436 May 28 21:12 1653743532833-4499-0-0.tsfile
-rw-rw-r-- 1 cluster cluster 1800 May 28 21:12
1653743532833-4499-0-0.tsfile.resource
-rw-rw-r-- 1 cluster cluster 14730240 May 28 21:13
1653743555186-*4500*-0-0.tsfile
-rw-rw-r-- 1 cluster cluster 1800 May 28 21:13
1653743555186-4500-0-0.tsfile.resource
-rw-rw-r-- 1 cluster cluster 46514354 May 30 02:54
1653750725577-*4801*-1-0.tsfile
-rw-rw-r-- 1 cluster cluster 1800 May 30 02:54
1653750725577-4801-1-0.tsfile.resource
-rw-rw-r-- 1 cluster cluster 46548041 May 30 02:46 1653751451969-4831-1-0.tsfile
-rw-rw-r-- 1 cluster cluster 1800 May 30 02:46
1653751451969-4831-1-0.tsfile.resource
-rw-rw-r-- 1 cluster cluster 46231288 May 30 01:56 1653752166635-4861-1-0.tsfile
-rw-rw-r-- 1 cluster cluster 1800 May 30 01:56
1653752166635-4861-1-0.tsfile.resource
> [ TTL ] Expired tsfiles with a TTL of more than 10 hours are not deleted
>
>
> Key: IOTDB-3040
> URL: https://issues.apache.org/jira/browse/IOTDB-3040
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Server
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Liuxuxin
>Priority: Major
> Labels: pull-request-available
> Attachments: config.properties, file status.png,
> iotdb-engine.properties, iotdb-env.sh, log-all-20220511.3.log.gz, log.png,
> logback.xml
>
>
> master_0427_b633df5
> 问题描述：
> 长测配置，设置TTL为24小时，
> grep ttl logs/log_all.log |tail -1
> 2022-04-29 08:57:07,404 [pool-11-IoTDB-TTL-Check-1] INFO
> o.a.i.d.e.s.DataRegion:1658 - Removed a file
> /data/iotdb_data/data/sequence/root.test.g_4/0/0/1651092583331-4021-1-0.tsfile{color:#DE350B}
> before Sat Sep 01 19:33:53 CST 2018 by ttl{color} (115392194000ms)
> ./tools/tsfileToolSet/print-tsfile-resource-files.sh
> 解析sg中第1个resource文件，查看时间戳范围，最大时间戳：
> device root.test.g_0.d_350, start time 1535749930200
> (2018-09-01T05:12:10.200+08:00[Asia/Shanghai]), {color:#DE350B}*end time
> 1535749948000 (2018-09-01T05:12:28+08:00[Asia/Shanghai])*{color}
> 有14个小时的过期文件没有被删除，磁盘空间无法释放。
> 1. 机器信息
> 私有云172.20.70.14/13 8C32G
> 数据库，bm配置文件见附件。
> 长测运行起来之后，24小时，设置TTL：
> #!/bin/bash
> #ttl默认大小24小时
> ttl=8640
> host=127.0.0.1
> port=6667
> function setTTLOfSg(){
> input_new_ttl=$1
> ./sbin/start-cli.sh -h $host -p $port -e "set ttl to root.** $input_new_ttl"
> }
> #计算操作系统当前时间和iotdb 序列max_time的差值
> function getDiffOfTime(){
># 操作系统当前值 秒
>os_now=`date +%s`
>max_time=`./sbin/start-cli.sh -h $host -p $port -e "show devices limit
> 1"|grep root|awk -F '|' '{print "./sbin/start-cli.sh -h " host " -p " port "
> -e \"select max_time(s_0) from " $2 " \" "}' host=$host port=$port|sh|grep "|
>"|awk -F '|' '{print $2}'`
>diffTime=$os_now*=1000))-max_time))
>echo $diffTime
> }
> # 2小时校正1次
> #sleep 24h
> for i in {1..4}
> do
>getDiffOfTime
>new_ttl=$((ttl+diffTime))
>echo $new_ttl
>setTTLOfSg $new_ttl
>sleep 2h
> done
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Reopened] (IOTDB-3164) Optimize memory allocation of the system and memory control of compaction
[ https://issues.apache.org/jira/browse/IOTDB-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 周沛辰 reopened IOTDB-3164: > Optimize memory allocation of the system and memory control of compaction > - > > Key: IOTDB-3164 > URL: https://issues.apache.org/jira/browse/IOTDB-3164 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: 周沛辰 >Assignee: 周沛辰 >Priority: Major > > The existing system memory allocation is prone to OOM problems at runtime, > and the memory allocation of compaction is unreasonable. See more details in > [https://apache-iotdb.feishu.cn/docs/doccnwoTDSPnxjVhRzbhqPbk1wh?from=from_copylink] -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3326) Fix device overlap between files after compaction in 0.12
周沛辰 created IOTDB-3326: -- Summary: Fix device overlap between files after compaction in 0.12 Key: IOTDB-3326 URL: https://issues.apache.org/jira/browse/IOTDB-3326 Project: Apache IoTDB Issue Type: Bug Reporter: 周沛辰 Assignee: 周沛辰 In 0.12, cross space compaction may result device overlap between target files, but timeseries does not overlap. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3327) [ create dataRegion ] Concurrency creates redundant dataRegion[s]
刘珍 created IOTDB-3327: - Summary: [ create dataRegion ] Concurrency creates redundant dataRegion[s] Key: IOTDB-3327 URL: https://issues.apache.org/jira/browse/IOTDB-3327 Project: Apache IoTDB Issue Type: Bug Components: mpp-cluster Reporter: 刘珍 master_0530_95884ad -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-3327) [ create dataRegion ] Concurrency creates redundant dataRegion[s]
[
https://issues.apache.org/jira/browse/IOTDB-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
刘珍 reassigned IOTDB-3327:
-
Assignee: Haonan Hou
> [ create dataRegion ] Concurrency creates redundant dataRegion[s]
> -
>
> Key: IOTDB-3327
> URL: https://issues.apache.org/jira/browse/IOTDB-3327
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Haonan Hou
>Priority: Major
> Attachments: screenshot-1.png
>
>
> master_0530_95884ad
> schema_replication_factor=1
> data_replication_factor=1
> 问题1：同1个sg的相同dataRegion被打印了3次（创建了3次？）
> 问题2：创建dataRegion的时候打印The data region root.test.g_9[10] is
> {color:red}*recovered *{color}successfully（海铭已知，海铭修改）
> !screenshot-1.png!
> 测试环境
> 192.168.130.2 （iotdb和bm） /home/benchmark/bm_20220322
> 配置文件见附件
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (IOTDB-3329) Fix overlap between pages and within one page after compaction of IOTDB before 0.13
周沛辰 created IOTDB-3329: -- Summary: Fix overlap between pages and within one page after compaction of IOTDB before 0.13 Key: IOTDB-3329 URL: https://issues.apache.org/jira/browse/IOTDB-3329 Project: Apache IoTDB Issue Type: Bug Reporter: 周沛辰 Assignee: 周沛辰 Before v0.13, cross space compaction may result overlap between pages and within one page in target files. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3332) [ ConfigNode ] Internal error processing getOrCreateSchemaPartition : NPE
刘珍 created IOTDB-3332: - Summary: [ ConfigNode ] Internal error processing getOrCreateSchemaPartition : NPE Key: IOTDB-3332 URL: https://issues.apache.org/jira/browse/IOTDB-3332 Project: Apache IoTDB Issue Type: Bug Components: mpp-cluster Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Yongzao Dan Attachments: ip62_confignode_log_error.log, ip62_iotdb-confignode.properties, ip62_iotdb-engine.properties, ip62_log_error.log, ip66_iotdb-confignode.properties, ip66_iotdb-engine.properties, ip68_bm_config.properties, ip68_iotdb-confignode.properties, ip68_iotdb-engine.properties master_0530_95884ad 3confignode，3datanode 并发创建schema，confignode抛异常： 2022-05-30 17:00:13,873 [pool-2-IoTDB-ConfigNodeRPC-Client-56] ERROR o.a.t.ProcessFunction:47 - Internal error processing getOrCreateSchemaPartition java.lang.NullPointerException: null at java.util.Collections.shuffle(Collections.java:457) at java.util.Collections.shuffle(Collections.java:427) at org.apache.iotdb.confignode.manager.load.balancer.allocator.CopySetRegionAllocator.genWeightedRandomRegion(CopySetRegionAllocator.java:114) at org.apache.iotdb.confignode.manager.load.balancer.allocator.CopySetRegionAllocator.allocateRegion(CopySetRegionAllocator.java:65) at org.apache.iotdb.confignode.manager.load.balancer.RegionBalancer.genRegionsAllocationPlan(RegionBalancer.java:87) at org.apache.iotdb.confignode.manager.load.LoadManager.initializeRegions(LoadManager.java:92) at org.apache.iotdb.confignode.manager.PartitionManager.checkAndAllocateRegionsIfNecessary(PartitionManager.java:315) at org.apache.iotdb.confignode.manager.PartitionManager.getOrCreateSchemaPartition(PartitionManager.java:122) at org.apache.iotdb.confignode.manager.ConfigManager.getOrCreateSchemaPartition(ConfigManager.java:367) at org.apache.iotdb.confignode.service.thrift.ConfigNodeRPCServiceProcessor.getOrCreateSchemaPartition(ConfigNodeRPCServiceProcessor.java:260) at org.apache.iotdb.confignode.rpc.thrift.ConfigIService$Processor$getOrCreateSchemaPartition.getResult(ConfigIService.java:1775) at org.apache.iotdb.confignode.rpc.thrift.ConfigIService$Processor$getOrCreateSchemaPartition.getResult(ConfigIService.java:1755) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 先看到datanode的异常，又去找confignode的日志的，datanode的异常： 2022-05-30 17:00:13,884 [pool-20-IoTDB-RPC-Client-54] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TException, message Fail to connect to any config node. Please check server it 2022-05-30 17:00:13,884 [pool-20-IoTDB-RPC-Client-47] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TException, message Fail to connect to any config node. Please check server it 2022-05-30 17:00:13,884 [pool-20-IoTDB-RPC-Client-34] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TException, message Fail to connect to any config node. Please check server it 2022-05-30 17:00:13,884 [pool-20-IoTDB-RPC-Client-38] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TException, message Fail to connect to any config node. Please check server it 2022-05-30 17:00:13,886 [pool-20-IoTDB-RPC-Client-48] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:65 - level-0 Exception class org.apache.thrift.TException, message Fail to connect to any config node. Please check server it 2022-05-30 17:00:13,888 [pool-20-IoTDB-RPC-Client-34] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Fail to connect to any config node. Please check server it, LocalizedMessage Fail to connect to any config node. Please check server it, org.apache.thrift.TException: Fail to connect to any config node. Please check server it at org.apache.iotdb.db.client.ConfigNodeClient.getOrCreateSchemaPartition(ConfigNodeClient.java:444) at org.apache.iotdb.db.client.ConfigNodeClient$$EnhancerByCGLIB$$7d184669.CGLIB$getOrCreateSchemaPartition$17() at org.apache.iotdb.db.client.ConfigNodeClient$$EnhancerByCGLIB$$7d184669$$FastClassByCGLIB$$e4527cc5.invoke() at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) at
[jira] [Created] (IOTDB-3336) [Optimize] Optimize some code about count nodes.
张洪胤 created IOTDB-3336: -- Summary: [Optimize] Optimize some code about count nodes. Key: IOTDB-3336 URL: https://issues.apache.org/jira/browse/IOTDB-3336 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3347) [ error message ] root.test.g_0 is not a legal path, because root.test.g_0 doesn't start with root.test.g_6
刘珍 created IOTDB-3347:
-
Summary: [ error message ] root.test.g_0 is not a legal path,
because root.test.g_0 doesn't start with root.test.g_6
Key: IOTDB-3347
URL: https://issues.apache.org/jira/browse/IOTDB-3347
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Yukun Zhou
Attachments: ip68_bm_config.properties
master_0530_95884ad
"show devices root.test.g_0" datanode error log:
2022-05-31 11:49:25,110 [Worker-Thread-3] ERROR
o.a.i.d.m.e.s.AbstractDriverThread:59 - Executor Worker-Thread-3 processes
failed
java.lang.RuntimeException:{color:red}* root.test.g_0 is not a legal path,
because root.test.g_0 doesn't start with root.test.g_6*{color}
at
org.apache.iotdb.db.mpp.execution.operator.schema.DevicesSchemaScanOperator.createTsBlock(DevicesSchemaScanOperator.java:62)
at
org.apache.iotdb.db.mpp.execution.operator.schema.SchemaQueryScanOperator.hasNext(SchemaQueryScanOperator.java:95)
at
org.apache.iotdb.db.mpp.execution.driver.Driver.processInternal(Driver.java:187)
at
org.apache.iotdb.db.mpp.execution.driver.Driver.lambda$processFor$1(Driver.java:127)
at
org.apache.iotdb.db.mpp.execution.driver.Driver.tryWithLock(Driver.java:274)
at
org.apache.iotdb.db.mpp.execution.driver.Driver.processFor(Driver.java:120)
at
org.apache.iotdb.db.mpp.execution.schedule.DriverTaskThread.execute(DriverTaskThread.java:57)
at
org.apache.iotdb.db.mpp.execution.schedule.AbstractDriverThread.run(AbstractDriverThread.java:55)
Caused by: org.apache.iotdb.commons.exception.IllegalPathException:
root.test.g_0 is not a legal path, because root.test.g_0 doesn't start with
root.test.g_6
at
org.apache.iotdb.db.metadata.mtree.traverser.Traverser.initStartIndexAndLevel(Traverser.java:120)
at
org.apache.iotdb.db.metadata.mtree.traverser.Traverser.(Traverser.java:89)
at
org.apache.iotdb.db.metadata.mtree.traverser.collector.CollectorTraverser.(CollectorTraverser.java:49)
at
org.apache.iotdb.db.metadata.mtree.traverser.collector.EntityCollector.(EntityCollector.java:38)
at
org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl$2.(MTreeBelowSGMemoryImpl.java:525)
at
org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl.getDevices(MTreeBelowSGMemoryImpl.java:525)
at
org.apache.iotdb.db.metadata.schemaregion.SchemaRegionMemoryImpl.getMatchedDevices(SchemaRegionMemoryImpl.java:986)
at
org.apache.iotdb.db.mpp.execution.operator.schema.DevicesSchemaScanOperator.createTsBlock(DevicesSchemaScanOperator.java:58)
... 7 common frames omitted
复现流程：
1. 3confignode，3datanode 192.168.10.62/66/68
2. bm 写入一些数据（配置文件见附件）
3. cli执行
./sbin/start-cli.sh -h 192.168.10.68 -e "show devices root.test.g_0"
客户端无异常，datanode 有error。
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
