[jira] [Created] (IOTDB-5029) Change merge statement to compact, and add debug info for compaction when executing this statement
Liuxuxin created IOTDB-5029: --- Summary: Change merge statement to compact, and add debug info for compaction when executing this statement Key: IOTDB-5029 URL: https://issues.apache.org/jira/browse/IOTDB-5029 Project: Apache IoTDB Issue Type: Improvement Components: Core/Server Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch In new cluster, the statement to trigger compaction in all datanodes is named `merge`, which is not proper. It's better to change it as `compact` or `trigger compaction`. What's more, it can be helpful to print some debug log of compaction selection when executing this statement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5028) Using `show timeseries` to query the schema in import-csv
Xuan Ronaldo created IOTDB-5028: --- Summary: Using `show timeseries` to query the schema in import-csv Key: IOTDB-5028 URL: https://issues.apache.org/jira/browse/IOTDB-5028 Project: Apache IoTDB Issue Type: Improvement Components: Client/CLI Reporter: Xuan Ronaldo Assignee: Xuan Ronaldo Fix For: master branch The import-csv tool is using `select * from ... limit 1` to query the schema. It will cost a lot of performance. Therefore, we need to use `show timeseries` to query the schema. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IOTDB-4971) dispatch write failed. status: TSStatus(code:506, subStatus:[]), message: null
[
https://issues.apache.org/jira/browse/IOTDB-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
刘珍 reopened IOTDB-4971:
---
> dispatch write failed. status: TSStatus(code:506, subStatus:[]), message: null
> --
>
> Key: IOTDB-4971
> URL: https://issues.apache.org/jira/browse/IOTDB-4971
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Jinrui Zhang
>Priority: Minor
> Attachments: del_ts.sh, down_delete_ts.conf, run_del_1.sh,
> run_del_2.sh, run_iotdb_4563.sh
>
>
> master_1117_d548214
> 1. start 3rep 3C 9D cluster
> 2. delete timeseries root.**和create metadata , write data 并发
> datanode （IP18） 有ERROR ：
> 2022-11-17 14:52:38,172
> [pool-24-IoTDB-DataNodeInternalRPC-Processor-17$20221117_065237_15126_11.1.0]
> {color:red}*ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:173 - dispatch
> write failed. status: TSStatus(code:506, subStatus:[]), message: null*{color}
> 复现流程
> 1. 启动3C9D集群
> 3C : 172.20.70.19/172.20.70.21/172.20.70.32
> 9D : 172.20.70.2/3/4/5/13/14/15/16/18
> 配置参数
> ConfigNode：
> MAX_HEAP_SIZE="8G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> cn_connection_timeout_ms=360
> Common ：
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> connection_timeout_ms=360
> max_connection_for_internal_service=200
> query_timeout_threshold=360
> schema_region_ratis_request_timeout_ms=180
> Datanode：
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> 2. 启动测试脚本
> down_delete_ts.conf放到${bm_dir}/conf下
> del_ts.sh 、run_del_1.sh 、 run_del_2.sh 、run_iotdb_4563.sh 这4个脚本放到${bm_dir}下
> 启动脚本是：run_iotdb_4563.sh
> 运行完成，查看 ip18datanode 日志。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-4971) dispatch write failed. status: TSStatus(code:506, subStatus:[]), message: null
[
https://issues.apache.org/jira/browse/IOTDB-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637584#comment-17637584
]
刘珍 commented on IOTDB-4971:
---
报错信息 dispatch write failed. status: TSStatus(code:506, subStatus:[]), message:
null中的null 能修改为PATH_ALREADY_EXIST对应的详细信息吗？
> dispatch write failed. status: TSStatus(code:506, subStatus:[]), message: null
> --
>
> Key: IOTDB-4971
> URL: https://issues.apache.org/jira/browse/IOTDB-4971
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Jinrui Zhang
>Priority: Minor
> Attachments: del_ts.sh, down_delete_ts.conf, run_del_1.sh,
> run_del_2.sh, run_iotdb_4563.sh
>
>
> master_1117_d548214
> 1. start 3rep 3C 9D cluster
> 2. delete timeseries root.**和create metadata , write data 并发
> datanode （IP18） 有ERROR ：
> 2022-11-17 14:52:38,172
> [pool-24-IoTDB-DataNodeInternalRPC-Processor-17$20221117_065237_15126_11.1.0]
> {color:red}*ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:173 - dispatch
> write failed. status: TSStatus(code:506, subStatus:[]), message: null*{color}
> 复现流程
> 1. 启动3C9D集群
> 3C : 172.20.70.19/172.20.70.21/172.20.70.32
> 9D : 172.20.70.2/3/4/5/13/14/15/16/18
> 配置参数
> ConfigNode：
> MAX_HEAP_SIZE="8G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> cn_connection_timeout_ms=360
> Common ：
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> connection_timeout_ms=360
> max_connection_for_internal_service=200
> query_timeout_threshold=360
> schema_region_ratis_request_timeout_ms=180
> Datanode：
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> 2. 启动测试脚本
> down_delete_ts.conf放到${bm_dir}/conf下
> del_ts.sh 、run_del_1.sh 、 run_del_2.sh 、run_iotdb_4563.sh 这4个脚本放到${bm_dir}下
> 启动脚本是：run_iotdb_4563.sh
> 运行完成，查看 ip18datanode 日志。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IOTDB-4972) [DispatchFailed] NPE at org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertNode.selfCheckDataTypes(InsertNode.java:251)
[
https://issues.apache.org/jira/browse/IOTDB-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yukun Zhou reassigned IOTDB-4972:
-
Assignee: Yukun Zhou (was: Jinrui Zhang)
> [DispatchFailed] NPE at
> org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertNode.selfCheckDataTypes(InsertNode.java:251)
> ---
>
> Key: IOTDB-4972
> URL: https://issues.apache.org/jira/browse/IOTDB-4972
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Yukun Zhou
>Priority: Major
> Attachments: image-2022-11-23-11-37-14-211.png, more_ts.conf
>
>
> master_1117_92c6a57
> 1. 启动3副本3C3D集群
> 2. benchmark创建元数据，写入数据
> 3.（ip62） datanode ERROR ，所有的数据写入失败（预期每个序列写入10个点，共5000万序列）：
> {color:red}*2022-11-17 16:32:59,456
> [pool-26-IoTDB-ClientRPC-Processor-35$20221117_083256_00512_3] ERROR
> o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:123 - [DispatchFailed]
> java.lang.NullPointerException: null
> at
> org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertNode.selfCheckDataTypes(InsertNode.java:251)*{color}
> at
> org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertTabletNode.validateAndSetSchema(InsertTabletNode.java:201)
> at
> org.apache.iotdb.db.mpp.plan.analyze.SchemaValidator.validate(SchemaValidator.java:64)
> at
> org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.executeDataInsert(RegionWriteExecutor.java:191)
> at
> org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitInsertTablet(RegionWriteExecutor.java:163)
> at
> org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitInsertTablet(RegionWriteExecutor.java:117)
> at
> org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertTabletNode.accept(InsertTabletNode.java:1086)
> at
> org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor.execute(RegionWriteExecutor.java:83)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchLocally(FragmentInstanceDispatcherImpl.java:232)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:137)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:119)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:90)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:106)
> at
> org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:287)
> at
> org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:205)
> at
> org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:150)
> at
> org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:164)
> at
> org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertTablet(ClientRPCServiceImpl.java:1234)
> at
> org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:4078)
> at
> org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:4058)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at
> org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 复现流程
> 1. 192.168.10.62/66/68 72C256GB 3C3D
> ConfigNode配置文件：
> MAX_HEAP_SIZE="8G"
> cn_connection_timeout_ms=360
> DataNode配置文件：
> MAX_HEAP_SIZE="192G"
> MAX_DIRECT_MEMORY_SIZE="32G"
> Common配置文件：
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> connection_timeout_ms=360
> max_connection_for_internal_service=1100
> enable_timed_flush_seq_memtable=true
> seq_memtable_flush_interval_in_ms=360
> seq_memtable_flush_check_interval_in_ms=60
>
[jira] [Commented] (IOTDB-4972) [DispatchFailed] NPE at org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertNode.selfCheckDataTypes(InsertNode.java:251)
[
https://issues.apache.org/jira/browse/IOTDB-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637554#comment-17637554
]
Jinrui Zhang commented on IOTDB-4972:
-
SENSOR_NUM is 10, which is too large.
Let's investigate why the NPE is threw with large amount of series.
> [DispatchFailed] NPE at
> org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertNode.selfCheckDataTypes(InsertNode.java:251)
> ---
>
> Key: IOTDB-4972
> URL: https://issues.apache.org/jira/browse/IOTDB-4972
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Jinrui Zhang
>Priority: Major
> Attachments: more_ts.conf
>
>
> master_1117_92c6a57
> 1. 启动3副本3C3D集群
> 2. benchmark创建元数据，写入数据
> 3.（ip62） datanode ERROR ，所有的数据写入失败（预期每个序列写入10个点，共5000万序列）：
> {color:red}*2022-11-17 16:32:59,456
> [pool-26-IoTDB-ClientRPC-Processor-35$20221117_083256_00512_3] ERROR
> o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:123 - [DispatchFailed]
> java.lang.NullPointerException: null
> at
> org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertNode.selfCheckDataTypes(InsertNode.java:251)*{color}
> at
> org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertTabletNode.validateAndSetSchema(InsertTabletNode.java:201)
> at
> org.apache.iotdb.db.mpp.plan.analyze.SchemaValidator.validate(SchemaValidator.java:64)
> at
> org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.executeDataInsert(RegionWriteExecutor.java:191)
> at
> org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitInsertTablet(RegionWriteExecutor.java:163)
> at
> org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor$WritePlanNodeExecutionVisitor.visitInsertTablet(RegionWriteExecutor.java:117)
> at
> org.apache.iotdb.db.mpp.plan.planner.plan.node.write.InsertTabletNode.accept(InsertTabletNode.java:1086)
> at
> org.apache.iotdb.db.mpp.execution.executor.RegionWriteExecutor.execute(RegionWriteExecutor.java:83)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchLocally(FragmentInstanceDispatcherImpl.java:232)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:137)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchWriteSync(FragmentInstanceDispatcherImpl.java:119)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatch(FragmentInstanceDispatcherImpl.java:90)
> at
> org.apache.iotdb.db.mpp.plan.scheduler.ClusterScheduler.start(ClusterScheduler.java:106)
> at
> org.apache.iotdb.db.mpp.plan.execution.QueryExecution.schedule(QueryExecution.java:287)
> at
> org.apache.iotdb.db.mpp.plan.execution.QueryExecution.start(QueryExecution.java:205)
> at
> org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:150)
> at
> org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:164)
> at
> org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertTablet(ClientRPCServiceImpl.java:1234)
> at
> org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:4078)
> at
> org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:4058)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at
> org.apache.iotdb.db.service.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:64)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 复现流程
> 1. 192.168.10.62/66/68 72C256GB 3C3D
> ConfigNode配置文件：
> MAX_HEAP_SIZE="8G"
> cn_connection_timeout_ms=360
> DataNode配置文件：
> MAX_HEAP_SIZE="192G"
> MAX_DIRECT_MEMORY_SIZE="32G"
> Common配置文件：
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> connection_timeout_ms=360
> max_connection_for_internal_service=1100
> enable_timed_flush_seq_memtable=true
> seq_memtable_flush_interval_in_ms=360
>
[jira] [Commented] (IOTDB-4632) [ MetaData ] org.apache.iotdb.db.exception.metadata.MeasurementAlreadyExistException: Path [xx] already exist
[ https://issues.apache.org/jira/browse/IOTDB-4632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637550#comment-17637550 ] 刘珍 commented on IOTDB-4632: --- master_1123_32e2f98 元数据路径已存在问题还存在，部分元数据创建失败 !screenshot-2.png! > [ MetaData ] > org.apache.iotdb.db.exception.metadata.MeasurementAlreadyExistException: Path > [xx] already exist > - > > Key: IOTDB-4632 > URL: https://issues.apache.org/jira/browse/IOTDB-4632 > Project: Apache IoTDB > Issue Type: Bug > Components: mpp-cluster >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Yukun Zhou >Priority: Major > Labels: pull-request-available > Attachments: more_ts.conf, screenshot-1.png, screenshot-2.png > > > master_1010_9917053 > schema region ： ratis > 3副本 > 500dev，10万sensor/dev , enable_auto_create_schema=false > 注册元数据，大量ERROR，部分dev没注册成功（ratis 超时？超时时间需可配置） > ERROR 信息： > 2022-10-13 11:37:36,676 > [192.168.10.62_50010@group-00020001-StateMachineUpdater] ERROR > o.a.i.d.m.v.SchemaExecutionVisitor:109 - IoTDB: MetaData error: > org.apache.iotdb.db.exception.metadata.MeasurementAlreadyExistException: Path > [root.test.g_1.d_221.s_48893] already exist > at > org.apache.iotdb.db.metadata.mtree.MTreeBelowSGMemoryImpl.createTimeseries(MTreeBelowSGMemoryImpl.java:218) > at > org.apache.iotdb.db.metadata.schemaregion.SchemaRegionMemoryImpl.createTimeseries(SchemaRegionMemoryImpl.java:622) > at > org.apache.iotdb.db.metadata.visitor.SchemaExecutionVisitor.visitCreateMultiTimeSeries(SchemaExecutionVisitor.java:106) > at > org.apache.iotdb.db.metadata.visitor.SchemaExecutionVisitor.visitCreateMultiTimeSeries(SchemaExecutionVisitor.java:63) > at > org.apache.iotdb.db.mpp.plan.planner.plan.node.metedata.write.CreateMultiTimeSeriesNode.accept(CreateMultiTimeSeriesNode.java:141) > at > org.apache.iotdb.db.consensus.statemachine.SchemaRegionStateMachine.write(SchemaRegionStateMachine.java:74) > at > org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.applyTransaction(ApplicationStateMachineProxy.java:135) > at > org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1588) > at > org.apache.ratis.server.impl.StateMachineUpdater.applyLog(StateMachineUpdater.java:239) > at > org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:182) > at java.lang.Thread.run(Thread.java:748) > 测试环境 > 1. 192.168.10.62/66/68 物理机 72CPU 256GB > bm在64 > ConfigNode > MAX_HEAP_SIZE="8G" > MAX_DIRECT_MEMORY_SIZE="8G" > > schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus > data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus > schema_replication_factor=3 > data_replication_factor=3 > connection_timeout_ms=120 > DataNode > MAX_HEAP_SIZE="192G" > MAX_DIRECT_MEMORY_SIZE="32G" > connection_timeout_ms=120 > max_connection_for_internal_service=1100 > enable_timed_flush_seq_memtable=true > seq_memtable_flush_interval_in_ms=360 > seq_memtable_flush_check_interval_in_ms=60 > enable_timed_flush_unseq_memtable=true > unseq_memtable_flush_interval_in_ms=360 > unseq_memtable_flush_check_interval_in_ms=60 > max_waiting_time_when_insert_blocked=360 > query_timeout_threshold=3600 > enable_auto_create_schema=false > 2. bm配置见附件 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5019) [write]data region leader write many wal files file after restarting datanode on it
[
https://issues.apache.org/jira/browse/IOTDB-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637548#comment-17637548
]
Haiming Zhu commented on IOTDB-5019:
This bug is caused by issue
[IOTDB-4896|https://issues.apache.org/jira/browse/IOTDB-4896], this issue skips
the memtable flush procedure and forgets to call memtable flush listener to
release wal.
> [write]data region leader write many wal files file after restarting datanode
> on it
> ---
>
> Key: IOTDB-5019
> URL: https://issues.apache.org/jira/browse/IOTDB-5019
> Project: Apache IoTDB
> Issue Type: Bug
>Reporter: changxue
>Assignee: Quan Siyi
>Priority: Major
> Attachments: empty memtable.jpeg, image-2022-11-22-16-30-27-046.png,
> image-2022-11-22-16-53-21-186.png
>
>
> [write]data region leader write many wal files after restarting datanode on it
> environment:
> 3C3D cluster, Nov. 21
> reproduction:
> 1. Using iotdb-benchmarks write data to iotdb cluster for more than 6 hours,
> only 1 device 1 sensor with double values. 2 replicas.
> 2. The 46 node failed to writing data, so I restart data node of it, and it's
> the data region leader
> 3. Continue writing data to the same timeseries for about 8 hours. I find
> that most of data lay on 44 node
> 问题：
> 1. 为什么重启46前，44，46节点上的数据分布还是很均衡的，重启46后，wal文件几乎就只写在44上了呢
> 2. 为什么写了那么多的wal文件，远远大于数据数量和size
>
> show regions(before and after restart datanode are the same):
>
> |RegionId|Type|Status|Database|SeriesSlotId|TimeSlotId|DataNodeId|Host|RpcPort|Role|
> |10|SchemaRegion|Running|root.aggr.g_0|1|0|1|172.20.70.44|6667|Follower|
> |10|SchemaRegion|Running|root.aggr.g_0|1|0|5|172.20.70.46|6667|Leader|
> |11|DataRegion|Running|root.aggr.g_0|1|10|1|172.20.70.44|6667|Follower|
> |11|DataRegion|Running|root.aggr.g_0|1|10|5|172.20.70.46|6667|Leader|
> iotdb-1: 44
> iotdb-2: 45
> iotdb-3: 46
> files:
> {code:java}
> atmos@i-rh6m726k root.aggr.g_0]$ ansible allnodes -m shell -a "find
> $IOTDB_HOME/data/datanode/data/sequence/root.aggr.g_0 -type f |wc -l"
> iotdb-1 | CHANGED | rc=0 >>
> 1694
> iotdb-2 | CHANGED | rc=0 >>
> 966
> iotdb-3 | CHANGED | rc=0 >>
> 183
> {code}
> !image-2022-11-22-16-53-21-186.png|width=895,height=348!
> monitor:
> !image-2022-11-22-16-30-27-046.png|width=870,height=629!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IOTDB-5019) [write]data region leader write many wal files file after restarting datanode on it
[
https://issues.apache.org/jira/browse/IOTDB-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haiming Zhu reassigned IOTDB-5019:
--
Assignee: Quan Siyi (was: Jinrui Zhang)
> [write]data region leader write many wal files file after restarting datanode
> on it
> ---
>
> Key: IOTDB-5019
> URL: https://issues.apache.org/jira/browse/IOTDB-5019
> Project: Apache IoTDB
> Issue Type: Bug
>Reporter: changxue
>Assignee: Quan Siyi
>Priority: Major
> Attachments: image-2022-11-22-16-30-27-046.png,
> image-2022-11-22-16-53-21-186.png
>
>
> [write]data region leader write many wal files after restarting datanode on it
> environment:
> 3C3D cluster, Nov. 21
> reproduction:
> 1. Using iotdb-benchmarks write data to iotdb cluster for more than 6 hours,
> only 1 device 1 sensor with double values. 2 replicas.
> 2. The 46 node failed to writing data, so I restart data node of it, and it's
> the data region leader
> 3. Continue writing data to the same timeseries for about 8 hours. I find
> that most of data lay on 44 node
> 问题：
> 1. 为什么重启46前，44，46节点上的数据分布还是很均衡的，重启46后，wal文件几乎就只写在44上了呢
> 2. 为什么写了那么多的wal文件，远远大于数据数量和size
>
> show regions(before and after restart datanode are the same):
>
> |RegionId|Type|Status|Database|SeriesSlotId|TimeSlotId|DataNodeId|Host|RpcPort|Role|
> |10|SchemaRegion|Running|root.aggr.g_0|1|0|1|172.20.70.44|6667|Follower|
> |10|SchemaRegion|Running|root.aggr.g_0|1|0|5|172.20.70.46|6667|Leader|
> |11|DataRegion|Running|root.aggr.g_0|1|10|1|172.20.70.44|6667|Follower|
> |11|DataRegion|Running|root.aggr.g_0|1|10|5|172.20.70.46|6667|Leader|
> iotdb-1: 44
> iotdb-2: 45
> iotdb-3: 46
> files:
> {code:java}
> atmos@i-rh6m726k root.aggr.g_0]$ ansible allnodes -m shell -a "find
> $IOTDB_HOME/data/datanode/data/sequence/root.aggr.g_0 -type f |wc -l"
> iotdb-1 | CHANGED | rc=0 >>
> 1694
> iotdb-2 | CHANGED | rc=0 >>
> 966
> iotdb-3 | CHANGED | rc=0 >>
> 183
> {code}
> !image-2022-11-22-16-53-21-186.png|width=895,height=348!
> monitor:
> !image-2022-11-22-16-30-27-046.png|width=870,height=629!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-5027) [ confignode ] The RegionRouteMap doesn't update after the leader Node is Unknown
刘珍 created IOTDB-5027:
-
Summary: [ confignode ] The RegionRouteMap doesn't update after
the leader Node is Unknown
Key: IOTDB-5027
URL: https://issues.apache.org/jira/browse/IOTDB-5027
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Yongzao Dan
Attachments: image-2022-11-23-10-26-35-175.png, log_confignode_all.log
m__600e376
1. 启动3副本3C3D集群，BM 写入查询 7*24小时
2. 验证各节点数据是否一致：只保留1节点在线，查询，查询报错，
报错是因为当前的查询路由策略是Leader优先级最高，但是show regions中Leader 有Unknown。
存在2个问题：
~ RegionRouteMap没更新。
~ 如果Leader 是Unknown，应该路由给Running的follower。
查询报错信息：
2022-11-23 10:03:07,158 [pool-80-IoTDB-ClientRPC-Processor-5] ERROR
o.a.i.d.u.ErrorHandlingUtils:90 - Status code: 310, Query Statement:
fetchResults failed
org.apache.iotdb.commons.exception.IoTDBException: can't connect to node
{}TEndPoint(ip:192.168.10.76, port:9003)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getResult(QueryExecution.java:394)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.getByteBufferBatchResult(QueryExecution.java:421)
at
org.apache.iotdb.db.utils.QueryDataSetUtils.convertQueryResultByFetchSize(QueryDataSetUtils.java:387)
at
org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.fetchResultsV2(ClientRPCServiceImpl.java:423)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$fetchResultsV2.getResult(IClientRPCService.java:3528)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$fetchResultsV2.getResult(IClientRPCService.java:3508)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
ConfigNode 日志：
!image-2022-11-23-10-26-35-175.png!
测试环境
https://apache-iotdb.feishu.cn/docs/doccn0blJR3zeJgEVB7PsGwyC5e#J8fSiJ
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-5026) Improvement of Last Query in MPP
Yuan Tian created IOTDB-5026: Summary: Improvement of Last Query in MPP Key: IOTDB-5026 URL: https://issues.apache.org/jira/browse/IOTDB-5026 Project: Apache IoTDB Issue Type: Improvement Components: Core/Query Reporter: Yuan Tian Assignee: liaolanyu In tsbs last-point case which contains 4000 devices and 10 sensors for each devuce, we found that too much time spent in plan phase while executing `select last * from root.cpu.*`. It's because even if we use aligned device in such case, we still treat each sensor as non-aligned, so we generate 40,000 plannodes which are too time-consuming. Actually, we should figure out a way to make control of the plan nodes number. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5018) Failed to COUNT group by level
[ https://issues.apache.org/jira/browse/IOTDB-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Minghui Liu reassigned IOTDB-5018: -- Assignee: Minghui Liu (was: 李伟豪) > Failed to COUNT group by level > -- > > Key: IOTDB-5018 > URL: https://issues.apache.org/jira/browse/IOTDB-5018 > Project: Apache IoTDB > Issue Type: Bug > Components: mpp-cluster >Reporter: Chao Wang >Assignee: Minghui Liu >Priority: Critical > Fix For: 1.0.0 > > > INSERT INTO root.test.wf01.wt01(timestamp, ts0, ts1, ts2, ts3, ts4, ts5) > VALUES (1, false, 3, 5, 6.8, 99.88, 'IoTDB'); > INSERT INTO root.test.wf01.wt01(timestamp, ts0, ts1, ts2, ts3, ts4, ts5) > VALUES (2, false, 3, 5, 6.8, 99.88, 'IoTDB'),(10, true, 3, 5, 6.8, 99.88, > 'IoTDB'); > INSERT INTO root.test.wf01.wt01(timestamp, ts0, ts1, ts2, ts3, ts4, ts5) > VALUES (11, false, 3, 5, 6.8, 99.88, 'IoTDB'),(20, false, 3, 5, 6.8, 99.88, > 'IoTDB'),(12, true, 3, 5, 6.8, 99.88, 'IoTDB'); > select COUNT(ts0), FIRST_VALUE(ts1), FIRST_VALUE(ts2) , LAST_VALUE(ts3) , > MAX_TIME(ts4), MIN_TIME(ts5) from root.test.wf01.wt01 group by level = 1,3 > > Msg: 500: [INTERNAL_SERVER_ERROR(500)] Exception occurred: "select COUNT(ts0) > from root.test.wf01.wt01 group by level = 1,3". executeStatement failed. null -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5013) Add start-standalone shell and stop-standalone shell.
[ https://issues.apache.org/jira/browse/IOTDB-5013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637535#comment-17637535 ] Jialin Qiao commented on IOTDB-5013: Update all start-server.sh in docs > Add start-standalone shell and stop-standalone shell. > - > > Key: IOTDB-5013 > URL: https://issues.apache.org/jira/browse/IOTDB-5013 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Gaofei Cao >Assignee: 陈哲涵 >Priority: Major > > Start 1C1D cluster -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5025) [Metric] Fix tag of metric in 0.13
张洪胤 created IOTDB-5025: -- Summary: [Metric] Fix tag of metric in 0.13 Key: IOTDB-5025 URL: https://issues.apache.org/jira/browse/IOTDB-5025 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 Fix same tags in metric of 0.13 and fix same category of metrics has different tag keys. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5024) [Metric] Fix tag of metrics
张洪胤 created IOTDB-5024: -- Summary: [Metric] Fix tag of metrics Key: IOTDB-5024 URL: https://issues.apache.org/jira/browse/IOTDB-5024 Project: Apache IoTDB Issue Type: Improvement Reporter: 张洪胤 Assignee: 张洪胤 There are some metrics in 0.14 version use same tag keys -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4473) Support TSBS Timeseries Benchmark
[ https://issues.apache.org/jira/browse/IOTDB-4473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637518#comment-17637518 ] Xingyu Liu commented on IOTDB-4473: --- 文档已补全 开发设计文档：https://apache-iotdb.feishu.cn/docx/Q4TydTObJofXTTxuF1Icuq4Knlh 用户使用手册：https://apache-iotdb.feishu.cn/docx/EQjFdAdL9oBvYqx0eKrcH0RYn6b > Support TSBS Timeseries Benchmark > - > > Key: IOTDB-4473 > URL: https://issues.apache.org/jira/browse/IOTDB-4473 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Julian Feinauer >Assignee: Xingyu Liu >Priority: Major > Attachments: image-2022-11-06-19-37-51-033.png, > image-2022-11-17-22-05-53-013.png > > > The tsbs Benchmark was initially created by InfluxDB and is now maintained by > timescale db and compares many state of the art timeseries databases: > [https://github.com/timescale/tsbs] > > IMHO we should add support for IoTDB -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5023) [0.14][cluster] Result of "count timeseries" is an int type
ZhangZhengming created IOTDB-5023: - Summary: [0.14][cluster] Result of "count timeseries" is an int type Key: IOTDB-5023 URL: https://issues.apache.org/jira/browse/IOTDB-5023 Project: Apache IoTDB Issue Type: Bug Reporter: ZhangZhengming Assignee: Yukun Zhou Attachments: image-2022-11-22-23-06-47-805.png It is recommended to change to long type. !image-2022-11-22-23-06-47-805.png|width=240,height=344! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5022) Windows jdk1.8.0_171: Unrecognized option: -Xlog:gc
[ https://issues.apache.org/jira/browse/IOTDB-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jialin Qiao reassigned IOTDB-5022: -- Assignee: 陈哲涵 > Windows jdk1.8.0_171: Unrecognized option: -Xlog:gc > --- > > Key: IOTDB-5022 > URL: https://issues.apache.org/jira/browse/IOTDB-5022 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.14.0-SNAPSHOT >Reporter: Jialin Qiao >Assignee: 陈哲涵 >Priority: Major > Attachments: image-2022-11-22-22-08-29-609.png, > image-2022-11-22-22-15-05-364.png > > > Failed to start IoTDB in windows with jdk1.8.0_171 > > !image-2022-11-22-22-15-05-364.png! > Unrecognized option: -Xlog:gc:C:\Users\zzm\Desktop\ab\gc.log > > !image-2022-11-22-22-08-29-609.png! > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5022) Windows jdk1.8.0_171: Unrecognized option: -Xlog:gc
[ https://issues.apache.org/jira/browse/IOTDB-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637282#comment-17637282 ] Jialin Qiao commented on IOTDB-5022: To temporally resolve： In datanode-env.bat Line 100: set IOTDB_HEAP_OPTS=-Xmx%MAX_HEAP_SIZE% -Xms%HEAP_NEWSIZE% -Xlog:gc:"%IOTDB_HOME%\gc.log" inplace by： set IOTDB_HEAP_OPTS=-Xmx%MAX_HEAP_SIZE% -Xms%HEAP_NEWSIZE% > Windows jdk1.8.0_171: Unrecognized option: -Xlog:gc > --- > > Key: IOTDB-5022 > URL: https://issues.apache.org/jira/browse/IOTDB-5022 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.14.0-SNAPSHOT >Reporter: Jialin Qiao >Priority: Major > Attachments: image-2022-11-22-22-08-29-609.png, > image-2022-11-22-22-15-05-364.png > > > Failed to start IoTDB in windows with jdk1.8.0_171 > > !image-2022-11-22-22-15-05-364.png! > Unrecognized option: -Xlog:gc:C:\Users\zzm\Desktop\ab\gc.log > > !image-2022-11-22-22-08-29-609.png! > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5022) Windows jdk1.8.0_171: Unrecognized option: -Xlog:gc
Jialin Qiao created IOTDB-5022: -- Summary: Windows jdk1.8.0_171: Unrecognized option: -Xlog:gc Key: IOTDB-5022 URL: https://issues.apache.org/jira/browse/IOTDB-5022 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.14.0-SNAPSHOT Reporter: Jialin Qiao Attachments: image-2022-11-22-22-08-29-609.png, image-2022-11-22-22-15-05-364.png Failed to start IoTDB in windows with jdk1.8.0_171 !image-2022-11-22-22-15-05-364.png! Unrecognized option: -Xlog:gc:C:\Users\zzm\Desktop\ab\gc.log !image-2022-11-22-22-08-29-609.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5021) Using Junit parameterized test to optimize schema IT framework
yanze chen created IOTDB-5021: - Summary: Using Junit parameterized test to optimize schema IT framework Key: IOTDB-5021 URL: https://issues.apache.org/jira/browse/IOTDB-5021 Project: Apache IoTDB Issue Type: Improvement Reporter: yanze chen Assignee: yanze chen In order to make the tests cover Schema_File mode and Memory mode, we originally needed to create two classes for each IT, such as IoTDBMetadataFetchIT and IoTDBMetadataFetchSchemaFileIT, which would result in a large number of classes. This ISSUE introduces a JUnit parametric test approach. An AbstractSchemaIT abstract class is created for all schema test, and its subclasses will automatically run tests in both schema engine modes. reference: [https://blog.csdn.net/w605283073/article/details/80051229] [https://stackoverflow.com/questions/27745691/how-to-combine-runwith-with-runwithparameterized-class] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4962) [Template] Delete timeseries of non-existent schema template without error
[
https://issues.apache.org/jira/browse/IOTDB-4962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17637204#comment-17637204
]
xiaozhihong commented on IOTDB-4962:
commit 66a23130b72f5e439c4ff49fe62608bd8b5205cc
!image-2022-11-22-19-38-05-358.png!
> [Template] Delete timeseries of non-existent schema template without error
> ---
>
> Key: IOTDB-4962
> URL: https://issues.apache.org/jira/browse/IOTDB-4962
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: xiaozhihong
>Assignee: Yukun Zhou
>Priority: Minor
> Attachments: image-2022-11-22-19-38-05-358.png
>
>
> In V0.14.0 version, delete timeseries of schema template t1 that not exist,
> but still can execute successfully without verification message.
> Like this:
>
> {code:java}
> IoTDB> show schema templates;
> ++
> |TemplateName|
> ++
> ++
> Empty set.
> It costs 0.002s
> IoTDB> delete timeseries of schema template t1 from root.ln.wf01.wt01;
> Msg: The statement is executed successfully.
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-5020) Throw NPE in CI testCreateSnapshotWithUnclosedTsFile
Jialin Qiao created IOTDB-5020: -- Summary: Throw NPE in CI testCreateSnapshotWithUnclosedTsFile Key: IOTDB-5020 URL: https://issues.apache.org/jira/browse/IOTDB-5020 Project: Apache IoTDB Issue Type: Bug Reporter: Jialin Qiao Attachments: image-2022-11-22-17-43-52-344.png https://github.com/apache/iotdb/actions/runs/3519541019/jobs/5899596610 Error: Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 32.374 s <<< FAILURE! - in org.apache.iotdb.db.engine.snapshot.IoTDBSnapshotTest [12241|https://github.com/apache/iotdb/actions/runs/3519541019/jobs/5899596610#step:6:12242]Error: testCreateSnapshotWithUnclosedTsFile(org.apache.iotdb.db.engine.snapshot.IoTDBSnapshotTest) Time elapsed: 15.801 s <<< ERROR! [12242|https://github.com/apache/iotdb/actions/runs/3519541019/jobs/5899596610#step:6:12243]java.lang.NullPointerException [12243|https://github.com/apache/iotdb/actions/runs/3519541019/jobs/5899596610#step:6:12244] at org.apache.iotdb.db.engine.snapshot.IoTDBSnapshotTest.tearDown(IoTDBSnapshotTest.java:60) !image-2022-11-22-17-43-52-344.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5019) [write]data region leader write less file after restarting datanode on it
changxue created IOTDB-5019:
---
Summary: [write]data region leader write less file after
restarting datanode on it
Key: IOTDB-5019
URL: https://issues.apache.org/jira/browse/IOTDB-5019
Project: Apache IoTDB
Issue Type: Bug
Reporter: changxue
Assignee: Jinrui Zhang
Attachments: image-2022-11-22-16-30-27-046.png
[write]data region leader write less file after restarting datanode on it
environment:
3C3D cluster, Nov. 21
reproduction:
1. Using iotdb-benchmarks write data to iotdb cluster for more than 6 hours,
only 1 device 1 sensor with double values. 2 replicas.
2. The 46 node failed to writing data, so I restart data node of it, and it's
the data region leader
3. Continue writing data to the same timeseries for about 8 hours. I find that
most of data lay on 44 node
show regions(2 replicas):
|RegionId|Type|Status|Database|SeriesSlotId|TimeSlotId|DataNodeId|Host|RpcPort|Role|
|10|SchemaRegion|Running|root.aggr.g_0|1|0|1|172.20.70.44|6667|Follower|
|10|SchemaRegion|Running|root.aggr.g_0|1|0|5|172.20.70.46|6667|Leader|
|11|DataRegion|Running|root.aggr.g_0|1|10|1|172.20.70.44|6667|Follower|
|11|DataRegion|Running|root.aggr.g_0|1|10|5|172.20.70.46|6667|Leader|
iotdb-1: 44
iotdb-2: 45
iotdb-3: 46
files:
{code}
atmos@i-rh6m726k root.aggr.g_0]$ ansible allnodes -m shell -a "find
$IOTDB_HOME/data/datanode/data/sequence/root.aggr.g_0 -type f |wc -l"
iotdb-1 | CHANGED | rc=0 >>
1694
iotdb-2 | CHANGED | rc=0 >>
966
iotdb-3 | CHANGED | rc=0 >>
183
{code}
monitor:
!image-2022-11-22-16-30-27-046.png|thumbnail!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
