[jira] [Created] (IOTDB-5139) [benchmark]1device over 3w sensors, insert nothing
changxue created IOTDB-5139: --- Summary: [benchmark]1device over 3w sensors, insert nothing Key: IOTDB-5139 URL: https://issues.apache.org/jira/browse/IOTDB-5139 Project: Apache IoTDB Issue Type: Bug Affects Versions: 1.0.0 Reporter: changxue Assignee: 张洪胤 Attachments: allnodes-log-3w.tar.gz, benchmark-logs-3w.log, benchmark-logs-50w.log, config.properties [benchmark]1device over 3w sensors, insert nothing environment: benchmark: 1.0 commit: 25c1f742 iotdb: 3C3D cluster, 1.0.0 release edition create timeseries succeeded but "show regions" showed schema info only. configs and logs see attachments. 问题： 1. 3万sensors的为什么没有数据写入。3千sensors的可以。我使用session.insertRecord自己写代码insert，是成功的。 2. loop=2, 第二次持续刷下面日志，就是不结束。 2022-12-07 14:56:44,540 INFO cn.edu.tsinghua.iot.benchmark.client.DataClient:137 - pool-2-thread-1 50.00% workload is done. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-2997) [ last query cache ] OOM
[ https://issues.apache.org/jira/browse/IOTDB-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644181#comment-17644181 ] 刘珍 commented on IOTDB-2997: --- 需先解决IOTDB-5131 再验证。 > [ last query cache ] OOM > > > Key: IOTDB-2997 > URL: https://issues.apache.org/jira/browse/IOTDB-2997 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Schema Manager >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Yukun Zhou >Priority: Major > Attachments: config.properties, image-2022-04-24-17-31-36-557.png, > image-2022-04-24-17-31-59-799.png, iotdb-engine.properties, iotdb-env.sh > > > master_0421_8db8876 > 问题描述： > DEVICE_NUMBER=50 > SENSOR_NUMBER=20, 共1000w个序列。 > CLIENT_NUMBER=10 > 写入顺序数据，90%写入，10% last 查询。 > MAX_HEAP_SIZE="20G" > MAX_DIRECT_MEMORY_SIZE="6G" > 其余参数默认，OOM > !image-2022-04-24-17-31-36-557.png! > jprofiler监控（bm的写入和查询已经kill掉）： > !image-2022-04-24-17-31-59-799.png! > 昊男帮忙定位，执行只有写入，没有查询。无OOM。 > 猜测last 查询在这种场景下没有控制内存 > iotdb配置文件，bm配置文件见附件。 > 机器信息：8C32G * 2 > 192.168.130.1 iotdb > 192.168.130.2 bm -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5039) [Load]An error is reported when loading a data file of about 100G
[
https://issues.apache.org/jira/browse/IOTDB-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644167#comment-17644167
]
Chao Wang commented on IOTDB-5039:
--
please reset longer dn_connection_timeout_ms in iotdb-common.properties and
restart iotdb
> [Load]An error is reported when loading a data file of about 100G
> -
>
> Key: IOTDB-5039
> URL: https://issues.apache.org/jira/browse/IOTDB-5039
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: xiaozhihong
>Assignee: yusicheng
>Priority: Major
> Labels: pull-request-available
> Attachments: logs.zip
>
>
> commit 8755c3fbeabe078ea7a4b1bb6e013db905d6e6a6
> Start 3C3D，1 replication.
> Enter CLi, execute sql :
> {code:java}
> Starting IoTDB Cli
> -
> _ _ __ __
> |_ _| | _ _ ||_ _ `.|_ _ \
> | | .--.|_/ | | \_| | | `. \ | |_) |
> | | / .'`\ \ | | | | | | | __'.
> _| |_| \__. | _| |_ _| |_.' /_| |__) |
> |_|'.__.' |_| |__.'|___/ version 0.14.0-SNAPSHOT (Build:
> 8755c3f)
> Successfully login at
> 172.20.31.6:6667
> IoTDB> load "/home/xzh/data"
> Msg: 1103: can't connect to node {}TEndPoint(ip:172.20.31.2, port:9003)
> IoTDB>
> {code}
> For logs information, please see the attachment.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-5043) take snapshot by sql for IoTDB upgrade
[ https://issues.apache.org/jira/browse/IOTDB-5043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644166#comment-17644166 ] QiangShaowei commented on IOTDB-5043: - 解决方案： 在升级重启时，让ConfigNode的执行计划都落盘。 从master回合两个共识相关配置，并修改默认值 # trigger a snapshot when snapshot_trigger_threshold logs are written # config_node_ratis_snapshot_trigger_threshold = 1 # preserve certain logs when take snapshot and purge # config_node_ratis_preserve_logs_num_when_purge = 0 > take snapshot by sql for IoTDB upgrade > -- > > Key: IOTDB-5043 > URL: https://issues.apache.org/jira/browse/IOTDB-5043 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: QiangShaowei >Assignee: QiangShaowei >Priority: Major > > IOTDB 1.0 重构了ConfigPlan相关的枚举类，导致之前装好的环境升到最新版启动不了，因为序列化和反序列化不一致。 > 需要手动触发task snapshot ,生成bin文件，然后提供转换工具，转换成新代码能解析的bin文件 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5138) Failed to load tsfile of 0.12 verson to 1.0 version
Chao Wang created IOTDB-5138:
Summary: Failed to load tsfile of 0.12 verson to 1.0 version
Key: IOTDB-5138
URL: https://issues.apache.org/jira/browse/IOTDB-5138
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 1.0.0
Reporter: Chao Wang
tsfile of 0.12 version load to 1.0 version .
2022-12-07 13:38:25,627 | ERROR | [20221207_053810_00097_5-211] | Dispatch
TsFileData error when parsing TsFile
/srv/aaa/BigData/iotdb/data1/data/sequence/root.group_2/0/0/1669957742562-1-1-1.tsfile.
| org.apache.iotdb.db.mpp.plan.scheduler.load.LoadTsFileScheduler
(LoadTsFileScheduler.java:152)
java.lang.IllegalStateException: Consume chunkData error, chunk offset:
2343823, measurement: s_231, chunkData: NonAlignedChunkData\{dataSize=20325,
timePartitionSlot=TTimePartitionSlot(startTime:153740160),
device='root.group_2.d_62', chunkHeader=CHUNK_HEADER{measurementID='s_231',
dataSize=1257, dataType=INT32, compressionType=SNAPPY, encodingType=PLAIN,
numOfPages=0, serializedSize=12}, needDecodeChunk=true}
at
org.apache.iotdb.db.engine.load.TsFileSplitter.consumeChunkData(TsFileSplitter.java:380)
at
org.apache.iotdb.db.engine.load.TsFileSplitter.splitTsFileByDataPartition(TsFileSplitter.java:229)
at
org.apache.iotdb.db.mpp.plan.scheduler.load.LoadTsFileScheduler.firstPhase(LoadTsFileScheduler.java:147)
at
org.apache.iotdb.db.mpp.plan.scheduler.load.LoadTsFileScheduler.start(LoadTsFileScheduler.java:131)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:250)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:190)
at
org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:181)
at
org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.executeStatement(ClientRPCServiceImpl.java:635)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatement.getResult(IClientRPCService.java:3122)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$executeStatement.getResult(IClientRPCService.java:3102)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at
org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:70)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2022-12-07 13:38:25,627 | INFO | [20221207_053810_00097_5-211] | Start
dispatching Load command for uuid 15ff11ef-3ce5-4452-8eee-e6a18e751233 |
org.apache.iotdb.db.mpp.plan.scheduler.load.LoadTsFileScheduler
(LoadTsFileScheduler.java:224)
2022-12-07 13:38:25,628 | ERROR | [20221207_053810_00097_5-211] | Dispatch
LoadCommand error to replicaSets
[TRegionReplicaSet(regionId:TConsensusGroupId(type:DataRegion, id:7),
dataNodeLocations:[TDataNodeLocation(dataNodeId:3,
clientRpcEndPoint:TEndPoint(ip:192.168.193.14, port:22260),
internalEndPoint:TEndPoint(ip:192.168.193.14, port:22250),
mPPDataExchangeEndPoint:TEndPoint(ip:192.168.193.14, port:22254),
dataRegionConsensusEndPoint:TEndPoint(ip:192.168.193.14, port:22252),
schemaRegionConsensusEndPoint:TEndPoint(ip:192.168.193.14, port:22253)),
TDataNodeLocation(dataNodeId:5, clientRpcEndPoint:TEndPoint(ip:192.168.193.7,
port:22260), internalEndPoint:TEndPoint(ip:192.168.193.7, port:22250),
mPPDataExchangeEndPoint:TEndPoint(ip:192.168.193.7, port:22254),
dataRegionConsensusEndPoint:TEndPoint(ip:192.168.193.7, port:22252),
schemaRegionConsensusEndPoint:TEndPoint(ip:192.168.193.7, port:22253)),
TDataNodeLocation(dataNodeId:4, clientRpcEndPoint:TEndPoint(ip:192.168.193.11,
port:22260), internalEndPoint:TEndPoint(ip:192.168.193.11, port:22250),
mPPDataExchangeEndPoint:TEndPoint(ip:192.168.193.11, port:22254),
dataRegionConsensusEndPoint:TEndPoint(ip:192.168.193.11, port:22252),
schemaRegionConsensusEndPoint:TEndPoint(ip:192.168.193.11, port:22253))])]
error. | org.apache.iotdb.db.mpp.plan.scheduler.load.LoadTsFileScheduler
(LoadTsFileScheduler.java:235)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (IOTDB-4450) [ StandAloneConsensus] Very low write performance at 100k timeseries per dev
[ https://issues.apache.org/jira/browse/IOTDB-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 刘珍 reopened IOTDB-4450: --- rel/1.0 1206_cbf7291 1副本1C1D 与 master_0421_8db8876 单机版对比 测试机器 fit 192.168.130.1 BM在192.168.130.2 /data1/bm_v1 （使用的配置文件iotdb_2997_lastquery.conf） > [ StandAloneConsensus] Very low write performance at 100k timeseries per dev > > > Key: IOTDB-4450 > URL: https://issues.apache.org/jira/browse/IOTDB-4450 > Project: Apache IoTDB > Issue Type: Bug > Components: mpp-cluster >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Yukun Zhou >Priority: Major > Attachments: 1_dev_10w_sensor.conf, > image-2022-09-19-18-21-11-615.png, ip72_datanode_logs.tar.gz, > ip74_logs.tar.gz, screenshot-1.png > > > m_0918_a28329e > 3C3D ，StandAloneConsensus，在单个设备包含的序列较多时，写入性能变得非常慢（写入超时，全部失败）。 > 1用户，1dev，10w序列，batch size=2，LOOP=10，即每序列写入20个点，相同数据库配置 > 与新单机的性能对比： > !image-2022-09-19-18-21-11-615.png! > ip2（bm连接的节点）error日志： > 2022-09-19 16:59:48,447 > [pool-25-IoTDB-ClientRPC-Processor-5$20220919_085928_4_3.1.0] ERROR > o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:190 - can't connect to node > TEndPoint(ip:192.168.10.74, port:9003) > org.apache.thrift.TException: Error in calling method sendPlanNode > at > org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94) > at > org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$b12b10d.sendPlanNode() > at > org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:177) > at > org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:145) > at > org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:125) > at > org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:96) > at > org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:101) > at > org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:264) > at > org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:184) > at > org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:146) > at > org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:160) > at > org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertTablet(ClientRPCServiceImpl.java:995) > at > org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:3512) > at > org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:3492) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > Caused by: org.apache.thrift.TException: Error in calling method > recv_sendPlanNode > at > org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94) > at > org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$b12b10d.recv_sendPlanNode() > at > org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Client.sendPlanNode(IDataNodeRPCService.java:442) > at > org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$b12b10d.CGLIB$sendPlanNode$123() > at > org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$b12b10d$$FastClassByCGLIB$$7b038c6e.invoke() > at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) > at > org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55) > ... 19 common frames omitted > Caused by: org.apache.thrift.TException: Error in calling method receiveBase > at > org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:94) > at >
[jira] [Assigned] (IOTDB-5119) [M4]There is an exception in the M4 function query
[
https://issues.apache.org/jira/browse/IOTDB-5119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei Rui reassigned IOTDB-5119:
--
Assignee: (was: Lei Rui)
> [M4]There is an exception in the M4 function query
> --
>
> Key: IOTDB-5119
> URL: https://issues.apache.org/jira/browse/IOTDB-5119
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Query, mpp-cluster
>Affects Versions: master branch, 1.0.0
>Reporter: xiaozhihong
>Priority: Major
> Attachments: image-2022-12-05-17-15-28-373.png
>
>
> In V1.0, Started 3C3D.
> Entered CLI, executed M4 query and reported a exception :
> {code:java}
> IoTDB> select temperature from root.ln.wf01.wt01;
> +-+-+
> | Time|root.ln.wf01.wt01.temperature|
> +-+-+
> |2017-11-05T10:20:00.000+08:00| 25.57756|
> |2017-11-05T10:21:00.000+08:00| 23.97946|
> |2017-11-05T10:22:00.000+08:00| 22.720444|
> +-+-+
> Total line number = 3
> It costs 0.007s
> IoTDB> select
> M4(temperature,'timeInterval'='1000','displayWindowBegin'='1','displayWindowEnd'='100')
> from root.ln.wf01.wt01;
> Msg: 301: Error occurred during executing UDTF#transform(RowWindow,
> PointCollector):
> java.lang.IndexOutOfBoundsException: Size is 0{code}
> The log is:
> !image-2022-12-05-17-15-28-373.png|width=542,height=225!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-4971) dispatch write failed. status: TSStatus(code:506, subStatus:[]), message: null
[
https://issues.apache.org/jira/browse/IOTDB-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644112#comment-17644112
]
刘珍 commented on IOTDB-4971:
---
上面是1条warn日志的内容，太多了，另外还有 ：message:null （最下面1行）
> dispatch write failed. status: TSStatus(code:506, subStatus:[]), message: null
> --
>
> Key: IOTDB-4971
> URL: https://issues.apache.org/jira/browse/IOTDB-4971
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Jinrui Zhang
>Priority: Minor
> Labels: pull-request-available
> Attachments: del_ts.sh, down_delete_ts.conf, run_del_1.sh,
> run_del_2.sh, run_iotdb_4563.sh
>
>
> master_1117_d548214
> 1. start 3rep 3C 9D cluster
> 2. delete timeseries root.**和create metadata , write data 并发
> datanode （IP18） 有ERROR ：
> 2022-11-17 14:52:38,172
> [pool-24-IoTDB-DataNodeInternalRPC-Processor-17$20221117_065237_15126_11.1.0]
> {color:red}*ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:173 - dispatch
> write failed. status: TSStatus(code:506, subStatus:[]), message: null*{color}
> 复现流程
> 1. 启动3C9D集群
> 3C : 172.20.70.19/172.20.70.21/172.20.70.32
> 9D : 172.20.70.2/3/4/5/13/14/15/16/18
> 配置参数
> ConfigNode：
> MAX_HEAP_SIZE="8G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> cn_connection_timeout_ms=360
> Common ：
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> connection_timeout_ms=360
> max_connection_for_internal_service=200
> query_timeout_threshold=360
> schema_region_ratis_request_timeout_ms=180
> Datanode：
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> 2. 启动测试脚本
> down_delete_ts.conf放到${bm_dir}/conf下
> del_ts.sh 、run_del_1.sh 、 run_del_2.sh 、run_iotdb_4563.sh 这4个脚本放到${bm_dir}下
> 启动脚本是：run_iotdb_4563.sh
> 运行完成，查看 ip18datanode 日志。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-5137) [ log level ][ SchemaRegion ] Change ERROR to WARN
刘珍 created IOTDB-5137: - Summary: [ log level ][ SchemaRegion ] Change ERROR to WARN Key: IOTDB-5137 URL: https://issues.apache.org/jira/browse/IOTDB-5137 Project: Apache IoTDB Issue Type: Bug Components: mpp-cluster Affects Versions: master branch, 1.0.0 Reporter: 刘珍 Assignee: Yukun Zhou rel/1.0 1206_cbf7291 master 下面ERROR 日志的级别需调为WARN 2022-12-07 11:13:07,866 [pool-26-IoTDB-DataNodeInternalRPC-Processor-7] ERROR o.a.i.d.m.e.e.RegionWriteExecutor$WritePlanNodeExecutionVisitor:363 - Metadata error: org.apache.iotdb.db.exception.metadata.MeasurementAlreadyExistException: Path [root.test3.g_9.d_269.s_5896] already exist at org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl.checkMeasurementExistence(MTreeBelowSGMemoryImpl.java:428) at org.apache.iotdb.db.metadata.schemaregion.SchemaRegionMemoryImpl.checkMeasurementExistence(SchemaRegionMemoryImpl.java:737) at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitCreateMultiTimeSeries(RegionWriteExecutor.java:353) at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitCreateMultiTimeSeries(RegionWriteExecutor.java:128) at org.apache.iotdb.db.mpp.plan.planner.plan.node.metedata.write.CreateMultiTimeSeriesNode.accept(CreateMultiTimeSeriesNode.java:155) at org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor.execute(RegionWriteExecutor.java:86) at org.apache.iotdb.db.service.thrift.impl.DataNodeInternalRPCServiceImpl.sendPlanNode(DataNodeInternalRPCServiceImpl.java:283) at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$sendPlanNode.getResult(IDataNodeRPCService.java:3607) at org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$sendPlanNode.getResult(IDataNodeRPCService.java:3587) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 复现流程同IOTDB-4971 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5136) NPE when use a null as username in session
[ https://issues.apache.org/jira/browse/IOTDB-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haonan Hou reassigned IOTDB-5136: - Assignee: 马子坤 > NPE when use a null as username in session > -- > > Key: IOTDB-5136 > URL: https://issues.apache.org/jira/browse/IOTDB-5136 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.13.3, 1.0.0 >Reporter: Haonan Hou >Assignee: 马子坤 >Priority: Minor > Attachments: 21151670378222_.pic.jpg, > image-2022-12-07-10-51-25-103.png > > > !21151670378222_.pic.jpg|width=545,height=117! > We should change the optional field username in client.thrift to a required > field. > !image-2022-12-07-10-51-25-103.png|width=483,height=247! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5136) NPE when use a null as username in session
Haonan Hou created IOTDB-5136: - Summary: NPE when use a null as username in session Key: IOTDB-5136 URL: https://issues.apache.org/jira/browse/IOTDB-5136 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.13.3, 1.0.0 Reporter: Haonan Hou Attachments: 21151670378222_.pic.jpg, image-2022-12-07-10-51-25-103.png !21151670378222_.pic.jpg|width=545,height=117! We should change the optional field username in client.thrift to a required field. !image-2022-12-07-10-51-25-103.png|width=483,height=247! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5135) Sync snapshot fails (also blocks ConfigNode snapshot) when sync file is empty.
Gaofei Cao created IOTDB-5135: - Summary: Sync snapshot fails (also blocks ConfigNode snapshot) when sync file is empty. Key: IOTDB-5135 URL: https://issues.apache.org/jira/browse/IOTDB-5135 Project: Apache IoTDB Issue Type: Bug Reporter: Gaofei Cao Assignee: yanze chen Attachments: image-2022-12-06-22-25-40-429.png Reproduce method: Turn down `config_node_ratis_snapshot_trigger_threshold` in iotdb-common.property Failure Reason: Rename returns false when tmpFile is empty. !image-2022-12-06-22-25-40-429.png|width=576,height=334! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5134) [Auth] Optimize the implementation of auth file
张洪胤 created IOTDB-5134: -- Summary: [Auth] Optimize the implementation of auth file Key: IOTDB-5134 URL: https://issues.apache.org/jira/browse/IOTDB-5134 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5133) [0.13] Export schema tool
yanze chen created IOTDB-5133: - Summary: [0.13] Export schema tool Key: IOTDB-5133 URL: https://issues.apache.org/jira/browse/IOTDB-5133 Project: Apache IoTDB Issue Type: New Feature Reporter: yanze chen Assignee: yanze chen Add an export schema function to export MTree and Tag from the current system into a non-deleted mlog.bin for loading mlog in a non-empty iotdb correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5132) create many timeseries with benchmark, failed with 301 null
changxue created IOTDB-5132:
---
Summary: create many timeseries with benchmark, failed with 301
null
Key: IOTDB-5132
URL: https://issues.apache.org/jira/browse/IOTDB-5132
Project: Apache IoTDB
Issue Type: Bug
Components: Core/Schema Manager
Affects Versions: 1.0.0
Reporter: changxue
Assignee: Gaofei Cao
Attachments: allnodes-log.tar.gz, config.properties, nohup.out
create many timeseries with benchmark, failed with 301 null
environment:
3C3D cluster, the 1.0.0 release bin of allinone
benchmark:
1.0 commit: 25c1f742
config see attachment of config.properties and nohup.out is its full logs.
I'm going to do performance testing with benchmark on IoTDB. There is a
scenario: create 50W timeseries on 1 device.
reproduction:
1. start IoTDB cluster successfully
2. 4 minutes later start the iot-benchmark
error log of benchmark:
{code}
2022-12-06 19:37:30,280 ERROR cn.edu.tsinghua.iot.benchmark.iotdb100.IoTDB:359
- Register IoTDB schema failed because
org.apache.iotdb.rpc.StatementExecutionException: 301: null
at org.apache.iotdb.rpc.RpcUtils.verifySuccess(RpcUtils.java:96)
at
org.apache.iotdb.session.SessionConnection.createAlignedTimeseries(SessionConnection.java:293)
at
org.apache.iotdb.session.Session.createAlignedTimeseries(Session.java:552)
at
cn.edu.tsinghua.iot.benchmark.iotdb100.IoTDB.registerTimeseries(IoTDB.java:332)
at
cn.edu.tsinghua.iot.benchmark.iotdb100.IoTDB.registerSchema(IoTDB.java:208)
at
cn.edu.tsinghua.iot.benchmark.tsdb.DBWrapper.registerSchema(DBWrapper.java:517)
at
cn.edu.tsinghua.iot.benchmark.client.SchemaClient.run(SchemaClient.java:94)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
error log of datanode:
{code}
2022-12-06 19:37:23,184 [grpc-default-executor-0] WARN
o.a.ratis.util.LogUtils:124 - 1: Failed requestVote 5->1#0
org.apache.ratis.protocol.exceptions.GroupMismatchException: 1:
group-0002 not found.
at
org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:150)
at
org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:351)
at
org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:360)
at
org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:355)
at
org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:618)
at
org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:175)
at
org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:382)
at
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
at
org.apache.ratis.thirdparty.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:354)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:866)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
说明：
1. 单独创建timeseries是没有问题的。
2. 猜测是并发问题
3. show timeseries root.** 发现并未创建成功，1个都没有
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-5131) [rel/1.0] 1rep1C1D， Follower is generated in the schema region, and "count devices" report an error
刘珍 created IOTDB-5131:
-
Summary: [rel/1.0] 1rep1C1D， Follower is generated in the schema
region, and "count devices" report an error
Key: IOTDB-5131
URL: https://issues.apache.org/jira/browse/IOTDB-5131
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 1.0.0
Reporter: 刘珍
Assignee: Yongzao Dan
Attachments: image-2022-12-06-19-08-02-946.png,
image-2022-12-06-19-08-38-711.png
rel/1.0 20221206
cbf7291
1. ./sbin/start-standalone.sh 启动1副本1C1D集群
2. BM运行附件中的配置
GROUP_NUMBER=10
DEVICE_NUMBER=50
SENSOR_NUMBER=20
BATCH_SIZE_PER_WRITE=1
LOOP=1000
3. {color:#DE350B}问题1 show regions 出现Follower
{color} !image-2022-12-06-19-08-02-946.png!
4. 问题2 count devices 立即返回错误
!image-2022-12-06-19-08-38-711.png!
后台 log
2022-12-06 18:55:56,256
[pool-32-IoTDB-MPPCoordinator-19$20221206_105556_71267_1.1.0] ERROR
o.a.i.d.m.e.e.RegionReadExecutor:54 - Execute FragmentInstance in
ConsensusGroup SchemaRegion[0] failed.
org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request
failed
at
org.apache.iotdb.consensus.ratis.RatisConsensus.read(RatisConsensus.java:323)
at
org.apache.iotdb.db.mpp.execution.executor.RegionReadExecutor.execute(RegionReadExecutor.java:46)
at
org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchLocally(FragmentInstanceDispatcherImpl.java:224)
at
org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:137)
at
org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchRead$0(FragmentInstanceDispatcherImpl.java:102)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ratis.protocol.exceptions.ServerNotReadyException:
1@group-0002 is not in [RUNNING]: current state is CLOSED
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$assertLifeCycleState$9(RaftServerImpl.java:733)
at
org.apache.ratis.util.LifeCycle.assertCurrentState(LifeCycle.java:253)
at
org.apache.ratis.server.impl.RaftServerImpl.assertLifeCycleState(RaftServerImpl.java:732)
at
org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:822)
at
org.apache.ratis.server.impl.RaftServerImpl.submitClientRequest(RaftServerImpl.java:944)
at
org.apache.ratis.server.impl.RaftServerProxy.submitClientRequest(RaftServerProxy.java:442)
at
org.apache.iotdb.consensus.ratis.RatisConsensus.read(RatisConsensus.java:318)
... 8 common frames omitted
测试环境
1. fit 机器 192.168.130.1 8C32GB
安装目录：/data1/relv1_1206_cbf7291
ConfigNode 配置
MAX_HEAP_SIZE="2G"
DataNode配置
MAX_HEAP_SIZE="20G"
MAX_DIRECT_MEMORY_SIZE="6G"
dn_wal_dirs=/data2/wal,/data3/wal
2. BM配置见附件
192.168.130.2 /data1/bm_v1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (IOTDB-4027) ERROR o.a.i.d.e.s.SnapshotLoader:94 - Exception occurs when creating links from snapshot directory to data directory
[ https://issues.apache.org/jira/browse/IOTDB-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 刘珍 reopened IOTDB-4027: --- rel/1.0 2022-12-06_cbf7291 ip6 follower 停datanode服务，5分钟后启动服务，同步失败： 2022-12-06 16:24:23,343 [grpc-default-executor-25] ERROR o.a.r.s.i.SnapshotInstallationHandler:96 - 7@group-00010004: installSnapshot failed java.nio.file.FileAlreadyExistsException: /data/iotdb/rel_1206_cbf7291_issue/data/datanode/data/snapshot/.tmp.group-00010004/snapshot-7b3e6a19-3184-4351-8d0f-fa996882675a/24_372652/sequence/root.ip4.g_0/4/2543/1670313965072-25-0-0.tsfile at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) at java.base/java.nio.file.Files.createDirectory(Files.java:690) at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:797) at java.base/java.nio.file.Files.createDirectories(Files.java:743) at org.apache.ratis.util.FileUtils.lambda$createDirectories$4(FileUtils.java:70) at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38) at org.apache.ratis.util.FileUtils.createDirectories(FileUtils.java:69) at org.apache.ratis.util.FileUtils.createDirectories(FileUtils.java:65) at org.apache.ratis.server.storage.SnapshotManager.installSnapshot(SnapshotManager.java:106) at org.apache.ratis.server.impl.ServerState.installSnapshot(ServerState.java:480) at org.apache.ratis.server.impl.SnapshotInstallationHandler.checkAndInstallSnapshot(SnapshotInstallationHandler.java:181) at org.apache.ratis.server.impl.SnapshotInstallationHandler.installSnapshotImpl(SnapshotInstallationHandler.java:120) at org.apache.ratis.server.impl.SnapshotInstallationHandler.installSnapshot(SnapshotInstallationHandler.java:94) at org.apache.ratis.server.impl.RaftServerImpl.installSnapshot(RaftServerImpl.java:1517) at org.apache.ratis.server.impl.RaftServerProxy.installSnapshot(RaftServerProxy.java:640) at org.apache.ratis.grpc.server.GrpcServerProtocolService$2.process(GrpcServerProtocolService.java:242) at org.apache.ratis.grpc.server.GrpcServerProtocolService$2.process(GrpcServerProtocolService.java:239) at org.apache.ratis.grpc.server.GrpcServerProtocolService$ServerRequestStreamObserver.onNext(GrpcServerProtocolService.java:124) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262) at org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:332) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:315) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:834) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 测试环境-私有云1期3副本3C9D 172.16.2.2~10 /data/iotdb/rel_1206_cbf7291_issue > ERROR o.a.i.d.e.s.SnapshotLoader:94 - Exception occurs when creating links > from snapshot directory to data directory > - > > Key: IOTDB-4027 > URL: https://issues.apache.org/jira/browse/IOTDB-4027 > Project: Apache IoTDB > Issue Type: Bug > Components: mpp-cluster >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Song Ziyang >Priority: Major > Labels: pull-request-available > Fix For: 0.14.0 > > Attachments: image-2022-08-03-09-39-10-230.png, > image-2022-08-03-09-39-48-739.png, image-2022-09-06-17-05-21-387.png, > ip18_befor_stop_datanode_log.tar.gz, ip18_restart_with-error_log.tar.gz, > ip4_2000_config.properties, screenshot-1.png > > > master_0801_55b5b17 > 问题描述 >
[jira] [Created] (IOTDB-5130) Accelerate the compaction of nonOverlap points in overlap pages
周沛辰 created IOTDB-5130: -- Summary: Accelerate the compaction of nonOverlap points in overlap pages Key: IOTDB-5130 URL: https://issues.apache.org/jira/browse/IOTDB-5130 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 In the fast compaction, the overlapping pages will be put into the pointPriorityReader, and the points are spit out in order. Every time a point is spit out, an element must be popped out from the priority queue, and then added into the queue. When the points do not overlap, it will cause a lot of invalid heap sort, which will reduce the compaction rate. For example: there are two pages, the time ranges are 1-500, 400-900, the actual overlapping part is only 400-500, and the rest of the points can be spit out directly, instead of adding into the queue and popping them out every time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5124) Examine whether the port is already in use when node start
[ https://issues.apache.org/jira/browse/IOTDB-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17643711#comment-17643711 ] Gaofei Cao commented on IOTDB-5124: --- Will be resolved in https://issues.apache.org/jira/browse/IOTDB-5125 > Examine whether the port is already in use when node start > -- > > Key: IOTDB-5124 > URL: https://issues.apache.org/jira/browse/IOTDB-5124 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Gaofei Cao >Assignee: Gaofei Cao >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5129) Exist two DataNode process when executed start-standalone.sh twice
Gaofei Cao created IOTDB-5129: - Summary: Exist two DataNode process when executed start-standalone.sh twice Key: IOTDB-5129 URL: https://issues.apache.org/jira/browse/IOTDB-5129 Project: Apache IoTDB Issue Type: Improvement Reporter: Gaofei Cao Assignee: Gaofei Cao -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IOTDB-2980) [ aligned timeseries performance ] High CPU usage & Low write performance
[ https://issues.apache.org/jira/browse/IOTDB-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 刘珍 reopened IOTDB-2980: --- rel/1.0 2022-12-06_cbf7291 写入性能 throughput(point/s) 对齐序列：16852681.29 普通序列：18227113.08 CPU使用率监控 > [ aligned timeseries performance ] High CPU usage & Low write performance > - > > Key: IOTDB-2980 > URL: https://issues.apache.org/jira/browse/IOTDB-2980 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Server >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Haonan Hou >Priority: Major > Attachments: aligned.properties, general.properties, > image-2022-04-22-15-12-11-013.png, screenshot-1.png > > > master_0421_8db8876 > 问题描述： > 在8C32G（fit机器）相同的客户端配置， > 对齐throughput(point/s) = 17358854.98， > 普通序列throughput(point/s)=18952049.86 > 对齐慢约9%，且cpu使用率高。 > !image-2022-04-22-15-12-11-013.png! > 测试步骤： > 1. 机器信息 > iotdb 192.168.130.1 /data1/master_0421_8db8876 > bm 192.168.130.2 /data1/benchmark_v13 > 2. 参数配置 > iotdb > MAX_HEAP_SIZE="20G" > MAX_DIRECT_MEMORY_SIZE="6G" > wal_dirs=/data2/wal,/data3/wal > benchmark参数配置见附件 > 3. 启动测试 > 监控机器资源，记录测试结果，对比。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
