[jira] [Created] (IOTDB-4562) Optimize ITimeIndex#getDevices()
yanze chen created IOTDB-4562: - Summary: Optimize ITimeIndex#getDevices() Key: IOTDB-4562 URL: https://issues.apache.org/jira/browse/IOTDB-4562 Project: Apache IoTDB Issue Type: Improvement Reporter: yanze chen Assignee: yanze chen -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4561) Improve the RowRecord computing speed in GroupByWithoutValueFilterDataSet by multi thread
[ https://issues.apache.org/jira/browse/IOTDB-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611326#comment-17611326 ] Guanfei Guo commented on IOTDB-4561: In my PC, with single thread strategy, the computation costs about 10.8 seconds. In contrast, it only costs about 3.7 seconds with multi thread strategy. > Improve the RowRecord computing speed in GroupByWithoutValueFilterDataSet by > multi thread > - > > Key: IOTDB-4561 > URL: https://issues.apache.org/jira/browse/IOTDB-4561 > Project: Apache IoTDB > Issue Type: Improvement > Components: Core/Server >Reporter: Guanfei Guo >Assignee: Guanfei Guo >Priority: Critical > Labels: pull-request-available > Fix For: 0.13.3 > > > In rel/0.13, the computation of a RowRecord in > GroupByWithoutValueFilterDataSet is a single thread process. In case the > aggregation result cotaining hundreds of columns, the time for these > computation is much long (in my PC, about 10.8 seconds), so I use multi > thread strategy to speed up the computation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4561) Improve the RowRecord computing speed in GroupByWithoutValueFilterDataSet by multi thread
Guanfei Guo created IOTDB-4561: -- Summary: Improve the RowRecord computing speed in GroupByWithoutValueFilterDataSet by multi thread Key: IOTDB-4561 URL: https://issues.apache.org/jira/browse/IOTDB-4561 Project: Apache IoTDB Issue Type: Improvement Components: Core/Server Reporter: Guanfei Guo Assignee: Guanfei Guo Fix For: 0.13.3 In rel/0.13, the computation of a RowRecord in GroupByWithoutValueFilterDataSet is a single thread process. In case the aggregation result cotaining hundreds of columns, the time for these computation is much long (in my PC, about 10.8 seconds), so I use multi thread strategy to speed up the computation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4525) when the data's size is large ,the IoTDB starts slowly
[ https://issues.apache.org/jira/browse/IOTDB-4525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17611314#comment-17611314 ] Haiming Zhu commented on IOTDB-4525: # There are too many tsfiles because compaction is interrupted # TimeIndex are degraded when system restarts, but the restart procedure needs reading some fields from DeviceTimeIndex, so we should recover these fileds before the degrade. > when the data's size is large ,the IoTDB starts slowly > --- > > Key: IOTDB-4525 > URL: https://issues.apache.org/jira/browse/IOTDB-4525 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: 刘俊男 >Assignee: Haiming Zhu >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png > > > 当数据量很大的时候，IoTDB启动很慢。 > 实验环数据量2.9T，data > 文件个数大约为80W，参数为默认参数。 > 文件个数 > !screenshot-1.png! > 执行结果 > !screenshot-2.png! > 每一个文件大小 > !screenshot-3.png! > 文件产生使用benchmark是用配置为 > Benchmark参数 > OP_INTERVAL=0 > POINT_STEP=1000 > OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0 > DEVICE_NUMBER=1 > SENSOR_NUMBER=100 > LOOP=1 > GROUP_NUMBER=20 > BIG_BATCH_SIZE=100 > 其他为默认参数，机器配置为4C 16G > 硬盘4.9T，堆内存为12G，操作为大约为执行时间60个小时候停掉数据库重启数据库。benchmark显示数据已经写入完毕 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4509) [cluster][grpc-default-worker-ELG-3-2]ERROR o.a.r.t.i.n.u.ResourceLeakDetector:319-LEAK: ByteBuf.release()
[
https://issues.apache.org/jira/browse/IOTDB-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610968#comment-17610968
]
Song Ziyang commented on IOTDB-4509:
recent gRPC community has memory leak:
[https://github.com/grpc/grpc-java/issues/9563]. It affects 1.48.1, and cause
OutOFDirectMemoryError. I' follow this issue until it is solved in gRPC.
> [cluster][grpc-default-worker-ELG-3-2]ERROR
> o.a.r.t.i.n.u.ResourceLeakDetector:319-LEAK: ByteBuf.release()
> --
>
> Key: IOTDB-4509
> URL: https://issues.apache.org/jira/browse/IOTDB-4509
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: FengQingxin
>Assignee: Song Ziyang
>Priority: Major
> Attachments: config.properties, confignode-env.sh, datanode-env.sh,
> iotdb-confignode.properties, iotdb-datanode.properties
>
>
> commit 7dc11bae567cb6aac18827b5c746e84538538ac3
> Author: ZhangHongYin <46039728+sprico...@users.noreply.github.com>
> Date: Thu Sep 22 23:36:17 2022 +0800
> IOTDB-4493 Fix null message in TSStatus (#7402)
>
>
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> Reproduce steps：
> # Setup a cluster with 3C3D({color:#de350b}MultiLeaderConsensus{color})
> # Using 3BMs to insert data(Batch=100)
> BM - 》 IoTDB Node
> 172.20.70.7 - 》 172.20.70.22
> 172.20.70.8 - 》 172.20.70.23
> 172.20.70.9 - 》 172.20.70.24
>
> 2022-09-23 09:02:30,026 [grpc-default-worker-ELG-3-2] ERROR
> o.a.r.t.i.n.u.ResourceLeakDetector:319 - LEAK: ByteBuf.release() was not
> called before it's garbage-collected. See
> [https://netty.io/wiki/reference-counted-objects.html] for more information.
> Recent access records:
> Created at:
>
> org.apache.ratis.thirdparty.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:401)
>
> org.apache.ratis.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
>
> org.apache.ratis.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
>
> org.apache.ratis.thirdparty.io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
>
> org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.expandCumulation(ByteToMessageDecoder.java:541)
>
> org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:97)
>
> org.apache.ratis.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:277)
>
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
>
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
>
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
>
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
>
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
>
> org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
>
> org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
>
> org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
>
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
>
> org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> java.lang.Thread.run(Thread.java:748)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-4560) Reconstruct configuration files
Jialin Qiao created IOTDB-4560: -- Summary: Reconstruct configuration files Key: IOTDB-4560 URL: https://issues.apache.org/jira/browse/IOTDB-4560 Project: Apache IoTDB Issue Type: Improvement Reporter: Jialin Qiao Assignee: Haiming Zhu Currently, we have iotdb-datanode.properties and iotdb-confignode.properties, however, we need to extract iotdb-common.properties. https://apache-iotdb.feishu.cn/docx/doxcnjzB2xd9sScIlkNYJUJA1zb -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-4559) Support Drop Trigger for ConfigNode in MPP
[ https://issues.apache.org/jira/browse/IOTDB-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 李伟豪 reassigned IOTDB-4559: -- Assignee: 李伟豪 > Support Drop Trigger for ConfigNode in MPP > -- > > Key: IOTDB-4559 > URL: https://issues.apache.org/jira/browse/IOTDB-4559 > Project: Apache IoTDB > Issue Type: Sub-task >Reporter: 李伟豪 >Assignee: 李伟豪 >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4559) Support Drop Trigger for ConfigNode in MPP
李伟豪 created IOTDB-4559: -- Summary: Support Drop Trigger for ConfigNode in MPP Key: IOTDB-4559 URL: https://issues.apache.org/jira/browse/IOTDB-4559 Project: Apache IoTDB Issue Type: Sub-task Reporter: 李伟豪 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4558) Use insertRowsOfOneDevice to execute insert multi rows sql statement
Haonan Hou created IOTDB-4558: - Summary: Use insertRowsOfOneDevice to execute insert multi rows sql statement Key: IOTDB-4558 URL: https://issues.apache.org/jira/browse/IOTDB-4558 Project: Apache IoTDB Issue Type: Improvement Reporter: Haonan Hou Currently, we use insertRowsStatement and insertRowsNode to execute insert multi rows sql like `insert into root.sg2.d1(time, s1, s2) values(1, 1, 1),(2, 2, 2), (3, 3, 3);` . It can be changed to insertRowsOfOneDeviceStatement and insertRowsOfOneDeviceNode. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4557) Memory used by PartitionCache is not restricted.
Jinrui Zhang created IOTDB-4557: --- Summary: Memory used by PartitionCache is not restricted. Key: IOTDB-4557 URL: https://issues.apache.org/jira/browse/IOTDB-4557 Project: Apache IoTDB Issue Type: Bug Reporter: Jinrui Zhang Assignee: 张洪胤 There are two issues here: # From current implementation, the total memory used by partition cache is not restricted. # The getPartition request for one same SeriesSlot may be sent more than one times when being invoked concurrently -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4556) [ remove datanode ] ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:287 - The consensus group DataRegion[24] doesn't exist
刘珍 created IOTDB-4556:
-
Summary: [ remove datanode ] ERROR
o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:287 - The consensus group
DataRegion[24] doesn't exist
Key: IOTDB-4556
URL: https://issues.apache.org/jira/browse/IOTDB-4556
Project: Apache IoTDB
Issue Type: Bug
Components: mpp-cluster
Affects Versions: 0.14.0-SNAPSHOT
Reporter: 刘珍
Assignee: Jinrui Zhang
m_0929_71d5f65
SchemaRegion : ratis
DataRegion : multiLeader
均为3副本，3C5D
启动客户端bm写入，缩容期间写入不停。
bm运行40分钟，缩容节点1（ip72），1小时38分钟缩容成功。
mv 节点1的data ，logs，再上线。
缩容节点2（ip73，开始缩容的时间09-29 14:10），此节点不包含DataRegion[24]
DataRegion[24]在ip74,ip75,ip76
但是ip73 error ：
2022-09-29 14:23:39,273
[pool-24-IoTDB-ClientRPC-Processor-2$20220929_062339_48081_4.1.0]
{color:#DE350B}*ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:287 - The
consensus group DataRegion[24] doesn't exist*{color}
2022-09-29 14:23:39,275 [MultiLeaderConsensusClientPool-selector-98] ERROR
o.a.i.c.m.l.IndexController:111 - {color:#DE350B}*failed to flush sync index.
cannot find previous version file. previous: 93500*{color}
2022-09-29 14:23:39,179 [pool-24-IoTDB-ClientRPC-Processor-45] WARN
o.a.i.d.u.ErrorHandlingUtils:62 - Status code: EXECUTE_STATEMENT_ERROR(400),
operation: insertTablet failed
java.lang.RuntimeException: org.apache.iotdb.commons.exception.IoTDBException:
There are no available RegionGroups currently, please check the status of
cluster DataNodes
at
org.apache.iotdb.db.mpp.plan.analyze.ClusterPartitionFetcher.getOrCreateDataPartition(ClusterPartitionFetcher.java:280)
at
org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitInsertTablet(AnalyzeVisitor.java:1236)
at
org.apache.iotdb.db.mpp.plan.analyze.AnalyzeVisitor.visitInsertTablet(AnalyzeVisitor.java:150)
at
org.apache.iotdb.db.mpp.plan.statement.crud.InsertTabletStatement.accept(InsertTabletStatement.java:121)
at
org.apache.iotdb.db.mpp.plan.statement.StatementVisitor.process(StatementVisitor.java:98)
at
org.apache.iotdb.db.mpp.plan.analyze.Analyzer.analyze(Analyzer.java:40)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.analyze(QueryExecution.java:236)
at
org.apache.iotdb.db.mpp.plan.execution.QueryExecution.(QueryExecution.java:138)
at
org.apache.iotdb.db.mpp.plan.Coordinator.createQueryExecution(Coordinator.java:100)
at
org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:133)
at
org.apache.iotdb.db.mpp.plan.Coordinator.execute(Coordinator.java:160)
at
org.apache.iotdb.db.service.thrift.impl.ClientRPCServiceImpl.insertTablet(ClientRPCServiceImpl.java:996)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:3512)
at
org.apache.iotdb.service.rpc.thrift.IClientRPCService$Processor$insertTablet.getResult(IClientRPCService.java:3492)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.iotdb.commons.exception.IoTDBException: There are no
available RegionGroups currently, please check the status of cluster DataNodes
... 20 common frames omitted
测试环境
1. 192.168.10.72/73/74/75/76 48CPU384GB
3C : 72,73,74
5D : 72 ,73,74,75,76
集群配置
ConfigNode
MAX_HEAP_SIZE="8G"
schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
schema_replication_factor=3
data_replication_factor=3
connection_timeout_ms=12
DataNode
MAX_HEAP_SIZE="256G"
MAX_DIRECT_MEMORY_SIZE="32G"
connection_timeout_ms=12
max_connection_for_internal_service=200
max_waiting_time_when_insert_blocked=60
query_timeout_threshold=3600
2. benchmark配置见附件
3. bm运行40分钟 缩容ip72
等待ip72 缩容完成，datanode进程退出
mv data logs
再次启动ip72
4. 缩容ip73 ，出现问题描述的ERROR 信息
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
