[jira] [Commented] (IOTDB-4405) ext-pipe kafka plugin trigger NullPointerException
[
https://issues.apache.org/jira/browse/IOTDB-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603909#comment-17603909
]
Bo Zhang commented on IOTDB-4405:
-
For resolving this problem, I did below analysis:
=
1) I analyzed the ext-pipe codes, and did not find root-cause.
2) I tested all 50 Tsfiles that were provided by SiChengYu and related member.
And these tsFiles can not trigger this exception.
3) I analyzed the codes of *PageHeader* related codes, find below issues.
code position: *org.apache.iotdb.tsfile.file.header.PageHeader#deserializeFrom*
!image-2022-09-14-14-42-08-582.png!
So, when 1 page is empty page, even if we call
{*}TsFileSequenceReader.readPageHeader(TSDataType, hasStatistic = true){*}, the
generated PageHeader will still have no *Statistic* info. This will affect
below codes:。
In codes ：{*}sync.datasource.TsFileOpBlock#getNonAlignedChunkPoints(){*}
*!image-2022-09-14-14-47-48-743.png!*
But because we can not re-produce this problem now, and current Tsfiles can not
re-produced this issue too.
So we are not sure whether this codes issue causes this problem.
Need futhre analysis.
> ext-pipe kafka plugin trigger NullPointerException
> --
>
> Key: IOTDB-4405
> URL: https://issues.apache.org/jira/browse/IOTDB-4405
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Others
>Reporter: Bo Zhang
>Assignee: Bo Zhang
>Priority: Major
> Attachments: image-2022-09-14-14-42-08-582.png,
> image-2022-09-14-14-47-48-743.png
>
>
> ext-pipe kafka plugin PR is: [https://github.com/apache/iotdb/pull/6688]
> The exception call-trace is as below:
> ===
> 6201 [pool-9-IoTDB-ExtPipePlugin-Worker-kafkasink-1] ERROR
> o.a.i.d.s.e.ExtPipePlugin - Unexpected system exception
> java.lang.NullPointerException: null
> at
> org.apache.iotdb.tsfile.file.header.PageHeader.getNumOfValues(PageHeader.java:105)
> at
> org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getNonAlignedChunkPoints(TsFileOpBlock.java:356)
> at
> org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getChunkPoints(TsFileOpBlock.java:582)
> at
> org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getOperation(TsFileOpBlock.java:691)
> at
> org.apache.iotdb.db.sync.datasource.PipeOpSgManager.getOperation(PipeOpSgManager.java:101)
> at
> org.apache.iotdb.db.sync.datasource.PipeOpManager.getOperation(PipeOpManager.java:173)
> at
> org.apache.iotdb.db.sync.externalpipe.ExtPipePlugin$DataTransmissionTask.call(ExtPipePlugin.java:496)
> at
> org.apache.iotdb.db.sync.externalpipe.ExtPipePlugin$DataTransmissionTask.call(ExtPipePlugin.java:312)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IOTDB-4405) ext-pipe kafka plugin trigger NullPointerException
[ https://issues.apache.org/jira/browse/IOTDB-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Zhang reassigned IOTDB-4405: --- Assignee: Bo Zhang > ext-pipe kafka plugin trigger NullPointerException > -- > > Key: IOTDB-4405 > URL: https://issues.apache.org/jira/browse/IOTDB-4405 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Others >Reporter: Bo Zhang >Assignee: Bo Zhang >Priority: Major > > 6201 [pool-9-IoTDB-ExtPipePlugin-Worker-kafkasink-1] ERROR > o.a.i.d.s.e.ExtPipePlugin - Unexpected system exception > java.lang.NullPointerException: null > at > org.apache.iotdb.tsfile.file.header.PageHeader.getNumOfValues(PageHeader.java:105) > at > org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getNonAlignedChunkPoints(TsFileOpBlock.java:356) > at > org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getChunkPoints(TsFileOpBlock.java:582) > at > org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getOperation(TsFileOpBlock.java:691) > at > org.apache.iotdb.db.sync.datasource.PipeOpSgManager.getOperation(PipeOpSgManager.java:101) > at > org.apache.iotdb.db.sync.datasource.PipeOpManager.getOperation(PipeOpManager.java:173) > at > org.apache.iotdb.db.sync.externalpipe.ExtPipePlugin$DataTransmissionTask.call(ExtPipePlugin.java:496) > at > org.apache.iotdb.db.sync.externalpipe.ExtPipePlugin$DataTransmissionTask.call(ExtPipePlugin.java:312) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:750) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4405) ext-pipe kafka plugin trigger NullPointerException
Bo Zhang created IOTDB-4405: --- Summary: ext-pipe kafka plugin trigger NullPointerException Key: IOTDB-4405 URL: https://issues.apache.org/jira/browse/IOTDB-4405 Project: Apache IoTDB Issue Type: Bug Components: Core/Others Reporter: Bo Zhang 6201 [pool-9-IoTDB-ExtPipePlugin-Worker-kafkasink-1] ERROR o.a.i.d.s.e.ExtPipePlugin - Unexpected system exception java.lang.NullPointerException: null at org.apache.iotdb.tsfile.file.header.PageHeader.getNumOfValues(PageHeader.java:105) at org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getNonAlignedChunkPoints(TsFileOpBlock.java:356) at org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getChunkPoints(TsFileOpBlock.java:582) at org.apache.iotdb.db.sync.datasource.TsFileOpBlock.getOperation(TsFileOpBlock.java:691) at org.apache.iotdb.db.sync.datasource.PipeOpSgManager.getOperation(PipeOpSgManager.java:101) at org.apache.iotdb.db.sync.datasource.PipeOpManager.getOperation(PipeOpManager.java:173) at org.apache.iotdb.db.sync.externalpipe.ExtPipePlugin$DataTransmissionTask.call(ExtPipePlugin.java:496) at org.apache.iotdb.db.sync.externalpipe.ExtPipePlugin$DataTransmissionTask.call(ExtPipePlugin.java:312) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4404) [ stop-datanode.sh ] DataNode Need to set read-only state
刘珍 created IOTDB-4404: - Summary: [ stop-datanode.sh ] DataNode Need to set read-only state Key: IOTDB-4404 URL: https://issues.apache.org/jira/browse/IOTDB-4404 Project: Apache IoTDB Issue Type: Bug Components: mpp-cluster Affects Versions: 0.14.0-SNAPSHOT Reporter: 刘珍 Assignee: Haiming Zhu Attachments: image-2022-09-14-13-41-46-600.png 3副本3C9D , stop-datanode.sh 会触发flush，但是并没有阻止新的写入（重启可能丢数据），需要设置read-only状态。stop-datanode.sh期间的日志（在flush后，还有create tsfile的操作）： !image-2022-09-14-13-41-46-600.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4403) Add retry interface for IStateMachine
[ https://issues.apache.org/jira/browse/IOTDB-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603882#comment-17603882 ] Song Ziyang commented on IOTDB-4403: refer to design doc: https://apache-iotdb.feishu.cn/docx/doxcnj0pH6Jm9pFWEEexa4opdXb > Add retry interface for IStateMachine > - > > Key: IOTDB-4403 > URL: https://issues.apache.org/jira/browse/IOTDB-4403 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Song Ziyang >Assignee: Song Ziyang >Priority: Major > Labels: pull-request-available > > To guarantee consistency in RatisConsensus, we should retry some failed apply > write result. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4403) Add retry interface for IStateMachine
Song Ziyang created IOTDB-4403: -- Summary: Add retry interface for IStateMachine Key: IOTDB-4403 URL: https://issues.apache.org/jira/browse/IOTDB-4403 Project: Apache IoTDB Issue Type: New Feature Reporter: Song Ziyang Assignee: Song Ziyang To guarantee consistency in RatisConsensus, we should retry some failed apply write result. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4402) Add FileLock to avoid multiple IoTDB processes access same data dir
Haonan Hou created IOTDB-4402: - Summary: Add FileLock to avoid multiple IoTDB processes access same data dir Key: IOTDB-4402 URL: https://issues.apache.org/jira/browse/IOTDB-4402 Project: Apache IoTDB Issue Type: Improvement Reporter: Haonan Hou If multiple IoTDB processes operate the same data dirs, the data file may be damaged. Need fileLock to protect the data dir. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-3624) Meet error when inserting data in SQL without timestamp
[
https://issues.apache.org/jira/browse/IOTDB-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603864#comment-17603864
]
xiaozhihong commented on IOTDB-3624:
I verified.
!image-2022-09-14-11-27-49-721.png|width=734,height=329!
> Meet error when inserting data in SQL without timestamp
> ---
>
> Key: IOTDB-3624
> URL: https://issues.apache.org/jira/browse/IOTDB-3624
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Server
>Affects Versions: 0.13.0, 0.14.0-SNAPSHOT
>Reporter: xiaozhihong
>Assignee: liuyu
>Priority: Minor
> Labels: pull-request-available
> Fix For: master branch
>
> Attachments: image-2022-06-23-10-43-49-858.png,
> image-2022-09-14-11-27-49-721.png
>
>
> The replay scenario is: There are two time series, one is text data type, the
> other is float data type. Default system time, inserted data at the same
> time, then there is a metadata error: the measurementList's size is not
> consistent with the valueList's size.
> {code:java}
> IoTDB> set storage group to root.sg;
> Msg: The statement is executed successfully.
> IoTDB> create timeseries root.sg.d1.s1 with datatype=FLOAT,encoding=RLE;
> Msg: The statement is executed successfully.
> IoTDB> create timeseries root.sg.d1.s2 with datatype=TEXT,encoding=RLE;
> Msg: 303: encoding RLE does not support TEXT
> IoTDB> create timeseries root.sg.d1.s2 with datatype=TEXT,encoding=PLAIN;
> Msg: The statement is executed successfully.
> IoTDB> show timeseries root.**;
> +-+-+-+++---++--+
> | timeseries|alias|storage
> group|dataType|encoding|compression|tags|attributes|
> +-+-+-+++---++--+
> |root.sg.d1.s1| null| root.sg| FLOAT| RLE| SNAPPY|null|
> null|
> |root.sg.d1.s2| null| root.sg| TEXT| PLAIN| SNAPPY|null|
> null|
> +-+-+-+++---++--+
> Total line number = 2
> It costs 0.072s
> IoTDB> insert into root.sg.d1(s1,s2) values(25,'test');
> Msg: 303: Check metadata error: the measurementList's size is not consistent
> with the valueList's size
> {code}
> The log picture is as follow:
> !image-2022-06-23-10-43-49-858.png|width=557,height=226!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IOTDB-4401) Region migrate process should be terminated when the datanode execute region migrate step failed
[ https://issues.apache.org/jira/browse/IOTDB-4401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaofei Cao reassigned IOTDB-4401: - Assignee: Gaofei Cao > Region migrate process should be terminated when the datanode execute region > migrate step failed > > > Key: IOTDB-4401 > URL: https://issues.apache.org/jira/browse/IOTDB-4401 > Project: Apache IoTDB > Issue Type: Bug >Reporter: Gaofei Cao >Assignee: Gaofei Cao >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4401) Region migrate process should be terminated when the datanode execute region migrate step failed
Gaofei Cao created IOTDB-4401: - Summary: Region migrate process should be terminated when the datanode execute region migrate step failed Key: IOTDB-4401 URL: https://issues.apache.org/jira/browse/IOTDB-4401 Project: Apache IoTDB Issue Type: Bug Reporter: Gaofei Cao -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4400) [new stand-alone]enableMetric, the write performance does not meet expectations
xiaozhihong created IOTDB-4400: -- Summary: [new stand-alone]enableMetric, the write performance does not meet expectations Key: IOTDB-4400 URL: https://issues.apache.org/jira/browse/IOTDB-4400 Project: Apache IoTDB Issue Type: Bug Components: Others Affects Versions: master branch, 0.14.0 Reporter: xiaozhihong Assignee: 张洪胤 Attachments: config.properties, image-2022-09-14-10-27-26-819.png commit 74fb350809b2f1488a90d6d7c420f27ec14b24e5 Turn on the monitoring function, and the two frameworks perform write performance tests for different metric levels. The final result is very confusing. Different levels or different frameworks, the performance of writing is not noticeable. Turn off monitoring, execute writing, and also no obvious difference is seen, and a positioning investigation needs to be done. Details: https://apache-iotdb.feishu.cn/docx/QUQSdbRaaoWDjQxcEz9cQjFdnsz?from=create_suite !image-2022-09-14-10-27-26-819.png|width=528,height=275! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4395) [cluster]Monitoring can support any disk information
[ https://issues.apache.org/jira/browse/IOTDB-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603842#comment-17603842 ] FengQingxin commented on IOTDB-4395: 是不是可以读取iotdb的配置文件获取data_dir？ > [cluster]Monitoring can support any disk information > > > Key: IOTDB-4395 > URL: https://issues.apache.org/jira/browse/IOTDB-4395 > Project: Apache IoTDB > Issue Type: Improvement > Components: Others >Reporter: xiaozhihong >Assignee: 张洪胤 >Priority: Minor > > In the current cluster version, only the information of the system disk in > the root directory can be supported, and other data disks or mounted disks > cannot be monitored. It is recommended to monitor the information of any disk. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IOTDB-3862) [shrink + Region-Migrate ] [pool-16-IoTDB-Region-Migrate-Pool-1] ERROR o.a.i.d.s.RegionMigrateService$RegionMigrateTask:349 - add new peer TEndPoint(ip:a.b.c.d, port:400
[
https://issues.apache.org/jira/browse/IOTDB-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
刘珍 reopened IOTDB-3862:
---
依赖最新ratis打包，基于master ba7bd5d0fd1a97807549bdb579c2557a3fbabbd9
问题还存在：
1. 192.168.130.3/4/5 3 副本3节点
bm在ip2 启动运行
2. 创建元数据成功，开始写入数据，扩容datanode ip1
3. ip5上分布有2个dataregion的leader（共3个dataregion）
缩容ip5，ip5的error log，{color:#DE350B}缩容后还会往ip5发查询语句？{color}
2022-09-13 16:44:33,412
[192.168.130.5_40010@group-00010005-SegmentedRaftLogWorker-flush-thread1]
ERROR o.a.r.s.r.s.BufferedWriteChannel:115 - Failed to flush channel
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110)
at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:379)
at
org.apache.ratis.server.raftlog.segmented.BufferedWriteChannel.fileChannelForce(BufferedWriteChannel.java:113)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-09-13 16:44:57,375 [20220913_084457_65466_5.1.0-142] ERROR
o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:166 - Execute FragmentInstance
failed: The consensus group SchemaRegion[1] doesn't exist
2022-09-13 16:44:57,597 [pool-22-IoTDB-ClientRPC-Processor-37] WARN
o.a.i.d.u.ErrorHandlingUtils:62 -{color:#DE350B} Status code:
INTERNAL_SERVER_ERROR(500), operation: "SELECT last s_50 FROM
root.test.g2_0.d_8". executeStatement failed {color}
2022-09-13 16:45:30,439 [pool-18-IoTDB-DataNodeInternalRPC-Processor-94] ERROR
o.a.i.d.s.t.i.DataNodeInternalRPCServiceImpl:628 - {color:#DE350B}*change
region DataRegion[3] leader failed
org.apache.iotdb.consensus.exception.RatisRequestFailedException: Ratis request
failed*{color}
at
org.apache.iotdb.consensus.ratis.RatisConsensus.transferLeader(RatisConsensus.java:526)
at
org.apache.iotdb.db.service.thrift.impl.DataNodeInternalRPCServiceImpl.transferLeader(DataNodeInternalRPCServiceImpl.java:619)
at
org.apache.iotdb.db.service.thrift.impl.DataNodeInternalRPCServiceImpl.changeRegionLeader(DataNodeInternalRPCServiceImpl.java:612)
at
org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$changeRegionLeader.getResult(IDataNodeRPCService.java:2643)
at
org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService$Processor$changeRegionLeader.getResult(IDataNodeRPCService.java:2623)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
exception
at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:92)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:233)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient.setConfiguration(GrpcClientProtocolClient.java:199)
at
org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:102)
at
org.apache.ratis.client.impl.BlockingImpl.sendRequest(BlockingImpl.java:139)
at
org.apache.ratis.client.impl.BlockingImpl.sendRequestWithRetry(BlockingImpl.java:104)
at
org.apache.ratis.client.impl.AdminImpl.setConfiguration(AdminImpl.java:46)
at
org.apache.ratis.client.api.AdminApi.setConfiguration(AdminApi.java:51)
at
org.apache.ratis.client.api.AdminApi.setConfiguration(AdminApi.java:40)
at
org.apache.iotdb.consensus.ratis.RatisConsensus.transferLeader(RatisConsensus.java:516)
... 10 common frames omitted
Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
UNAVAILABLE: io exception
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at
org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.setConfiguration(AdminProtocolServiceGrpc.java:490)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient.lambda$setConfiguration$6(GrpcClientProtocolClient.java:201)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientP
[GitHub] [iotdb-client-go] HTHou merged pull request #51: [To rel/0.13] support create aligned timeseries
HTHou merged PR #51: URL: https://github.com/apache/iotdb-client-go/pull/51 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (IOTDB-4399) Control chunk size for memtable
Quan Siyi created IOTDB-4399: Summary: Control chunk size for memtable Key: IOTDB-4399 URL: https://issues.apache.org/jira/browse/IOTDB-4399 Project: Apache IoTDB Issue Type: Improvement Reporter: Quan Siyi Assignee: Quan Siyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4257) Design and implement ReadChunkCompactionMemoryEstimator
[ https://issues.apache.org/jira/browse/IOTDB-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603831#comment-17603831 ] Liuxuxin commented on IOTDB-4257: - Design doc: https://apache-iotdb.feishu.cn/docx/doxcnwoL4ROgQ37l2yIf6r99ffh > Design and implement ReadChunkCompactionMemoryEstimator > --- > > Key: IOTDB-4257 > URL: https://issues.apache.org/jira/browse/IOTDB-4257 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: master branch, 0.13.2 > > > Currently there is memory estimator for cross space compaction, and a memory > control framework for compaction. We need to design and implement -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-3455) Make data_region_num take effect in new standalone
[ https://issues.apache.org/jira/browse/IOTDB-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603830#comment-17603830 ] Liuxuxin commented on IOTDB-3455: - Writing with 1-vsg !image-2022-09-14-09-04-41-108.png! Writing with 3 vsg: !image-2022-09-14-09-04-53-244.png! > Make data_region_num take effect in new standalone > -- > > Key: IOTDB-3455 > URL: https://issues.apache.org/jira/browse/IOTDB-3455 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Assignee: Liuxuxin >Priority: Major > Labels: pull-request-available > Attachments: image-2022-09-14-09-04-30-833.png, > image-2022-09-14-09-04-41-108.png, image-2022-09-14-09-04-53-244.png > > > ConfigNode is a process used in the new cluster. To make the processing logic > clear, we need to fake a ConfigNode in the new standalone version, called > LocalConfigNode, which is a module but not a process. > > Currently, in the new Standalone version, we just allocate one DataRegion and > one SchemaRegion for each Storage Group. This means the data_region_num does > not take effect. > > We need to let data_region_num take effect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4398) Migrate test cases to SessionIT
伊丹翔 created IOTDB-4398: -- Summary: Migrate test cases to SessionIT Key: IOTDB-4398 URL: https://issues.apache.org/jira/browse/IOTDB-4398 Project: Apache IoTDB Issue Type: New Feature Reporter: 伊丹翔 Assignee: 伊丹翔 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4397) Add Thread Model Doc for RatisConsensus
Song Ziyang created IOTDB-4397: -- Summary: Add Thread Model Doc for RatisConsensus Key: IOTDB-4397 URL: https://issues.apache.org/jira/browse/IOTDB-4397 Project: Apache IoTDB Issue Type: New Feature Reporter: Song Ziyang Assignee: Song Ziyang Provide runtime execution thread model for RatisConsensus. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [iotdb-client-go] HTHou merged pull request #50: add create aligned timeseries request
HTHou merged PR #50: URL: https://github.com/apache/iotdb-client-go/pull/50 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (IOTDB-4367) [MultiLeaderConsensus] thread leak
[
https://issues.apache.org/jira/browse/IOTDB-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603548#comment-17603548
]
Jinrui Zhang commented on IOTDB-4367:
-
We need to confirm whether this issue is led by dead-lock or resource-lack
> [MultiLeaderConsensus] thread leak
> --
>
> Key: IOTDB-4367
> URL: https://issues.apache.org/jira/browse/IOTDB-4367
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Jinrui Zhang
>Priority: Major
> Attachments: image-2022-09-08-18-17-19-185.png, more_metadata.conf
>
>
> m_0908_7915b3f ，3副本3C3D
> benchmark 300用户并发，3000设备，每个设备1万个序列，
> 往每个序列{color:#DE350B}写入1个点{color}，出现{color:#DE350B}线程泄露（IoTDB-DataNodeInternalRPC-Processor）{color}，且3个节点都有{color:#DE350B}乱序文件{color}：
> !image-2022-09-08-18-17-19-185.png!
> 田原查看日志/stack（benchmark连72）：
> 72往73和74转发写入请求，然后73和74处理不过来了，很多这些写入请求在73和74上阻塞住了；然后72上这个转发的线程超时（72和73上的处理线程不会结束，会一直等到处理完成），就返回给benchmark说写入失败了，benchmark的写入线程记录下这个失败后，就会开始进行下一个写入；72接收到新写入后，又会不断往73和74发，又会在73和74上创建新的处理写入的线程。
> 复现流程：
> 1. 192.168.10.72/ 73/74 48核384G
> benchmark 在71
> 2. 集群参数
> confignode
> MAX_HEAP_SIZE="8G"
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> datanode
> MAX_HEAP_SIZE="256G"
> MAX_DIRECT_MEMORY_SIZE="32G"
> {color:#DE350B}max_connection_for_internal_service=300{color}
> enable_timed_flush_seq_memtable=true
> seq_memtable_flush_interval_in_ms=60
> seq_memtable_flush_check_interval_in_ms=30
> enable_timed_flush_unseq_memtable=true
> unseq_memtable_flush_interval_in_ms=60
> unseq_memtable_flush_check_interval_in_ms=30
> max_waiting_time_when_insert_blocked=360
> query_timeout_threshold=360
> 3. benchmark配置见附件
> 运行起来，创建元数据成功，写入数据会有报错，就可以复现，bm端：
> 2022-09-08 18:05:05,571 ERROR
> cn.edu.tsinghua.iotdb.benchmark.tsdb.DBWrapper:131 - Insert batch failed
> because
> org.apache.iotdb.rpc.StatementExecutionException: 400:
> [EXECUTE_STATEMENT_ERROR(400)] Exception occurred: insertTablet failed.
> org.apache.iotdb.commons.exception.IoTDBException: some children of root.test
> have already been set to storage group
> at org.apache.iotdb.rpc.RpcUtils.verifySuccess(RpcUtils.java:94)
> at
> org.apache.iotdb.rpc.RpcUtils.verifySuccessWithRedirection(RpcUtils.java:115)
> at
> org.apache.iotdb.session.SessionConnection.insertTablet(SessionConnection.java:589)
> at org.apache.iotdb.session.Session.insertTablet(Session.java:1573)
> at org.apache.iotdb.session.Session.insertTablet(Session.java:1560)
> at
> cn.edu.tsinghua.iotdb.benchmark.iotdb013.IoTDBSession.insertOneBatchByTablet(IoTDBSession.java:168)
> at
> cn.edu.tsinghua.iotdb.benchmark.iotdb013.IoTDBSessionBase.insertOneBatch(IoTDBSessionBase.java:156)
> at
> cn.edu.tsinghua.iotdb.benchmark.tsdb.DBWrapper.insertOneBatch(DBWrapper.java:83)
> at
> cn.edu.tsinghua.iotdb.benchmark.client.generate.GenerateDataMixClient.ingestionOperation(GenerateDataMixClient.java:135)
> at
> cn.edu.tsinghua.iotdb.benchmark.client.generate.GenerateDataMixClient.doTest(GenerateDataMixClient.java:50)
> at
> cn.edu.tsinghua.iotdb.benchmark.client.DataClient.run(DataClient.java:144)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-4380) The log dispatcher is inconsistent with the search index of the wal node
[
https://issues.apache.org/jira/browse/IOTDB-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603547#comment-17603547
]
Jinrui Zhang commented on IOTDB-4380:
-
[~SpriCoder] Let's confirm whether following scenario would happen or not.
The safeIndex is 2958 and the entry whose index is less than 2958 are all
deleted in WAL files.
If yes, this bug would happen
> The log dispatcher is inconsistent with the search index of the wal node
>
>
> Key: IOTDB-4380
> URL: https://issues.apache.org/jira/browse/IOTDB-4380
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Haiming Zhu
>Priority: Major
> Attachments: ip74_logs.tar.gz, more_metadata.conf
>
>
> m_0908_7915b3f。
> 问题描述
> {color:#DE350B}*log dispatcher与wal node的search index不一致 ，
> datanode重启成功后日志一直刷*{color}：
> 2022-09-09 16:32:00,011 [pool-33-IoTDB-LogDispatcher-DataRegion[12]-2] INFO
> o.a.i.d.w.n.WALNode$PlanNodeIterator:695 - timeout when waiting for next WAL
> entry ready, execute rollWALFile. {color:#DE350B}*Current search index in wal
> buffer is 2959, and next target index is 2501 *{color}
> MultiLeaderConsensus，3副本3节点
> 1. 创建元数据过程中，kill ip74 的datanode PID
> benchmark配置文件见附件。
> 2. 清空ip74 的操作系统缓存，启动ip74的datanode
> 3. 再次重新运行benchmark同一配置，IS_DELETE_DATA=true
> 这个参数为true，会先执行delete storage group root.test.*;
> benchmark运行完成，stop ip74的datanode服务
> 备份data 为/data/mpp_test/m_0908_7915b3f/datanode/data_for_recovery_Test
> 4. 清ip74操作系统缓存，启动datanode服务
> 再次运行benchmark同一配置，benchmark运行完成，
> 查看ip74的日志，看到
> 2022-09-09 15:43:13,691 [pool-23-IoTDB-MPPDataExchangeRPC-Processor-40] ERROR
> o.a.t.ProcessFunction:47 - Internal error processing getDataBlock
> org.apache.thrift.TException: Source fragment instance not found. Fragment
> instance ID: TFragmentInstanceId(queryId:20220909_074205_19400_3,
> fragmentId:2, instanceId:0).
> at
> org.apache.iotdb.db.mpp.execution.exchange.MPPDataExchangeManager$MPPDataExchangeServiceImpl.getDataBlock(MPPDataExchangeManager.java:90)
> at
> org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Processor$getDataBlock.getResult(MPPDataExchangeService.java:326)
> at
> org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Processor$getDataBlock.getResult(MPPDataExchangeService.java:306)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2022-09-09 15:43:15,312 [20220909_074205_19400_3.2.0.SinkHandle-3074] ERROR
> o.a.i.d.m.e.e.SinkHandle:281 - The TsBlock doesn't exist. Sequence ID is 1,
> remaining map is
> [0=]
> 2022-09-09 15:43:17,119 [pool-23-IoTDB-MPPDataExchangeRPC-Processor-22] ERROR
> o.a.t.ProcessFunction:47 - Internal error processing getDataBlock
> java.lang.IllegalStateException: The data block doesn't exist. Sequence ID: 1
> at
> org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.getSerializedTsBlock(SinkHandle.java:285)
> at
> org.apache.iotdb.db.mpp.execution.exchange.MPPDataExchangeManager$MPPDataExchangeServiceImpl.getDataBlock(MPPDataExchangeManager.java:97)
> at
> org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Processor$getDataBlock.getResult(MPPDataExchangeService.java:326)
> at
> org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Processor$getDataBlock.getResult(MPPDataExchangeService.java:306)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 5. 停止ip74的datanode服务
> 备份data 到/data/mpp_test/m_0908_7915b3f/datanode/data_for_recovery_Test_2
> 清ip74操作系统缓存，启动ip74的datanode ，失败：
> 2022-09-09 16:44:00,039 [pool-33-IoTDB-LogDispatcher-DataRegion[12]-2] INFO
> o.a.i.d.w.n.WALNode$PlanNodeIterator:695 - timeout when waiting for next WAL
> entry ready, execute rollWALFile. Current search index in wal buffer is 2959,
> and next
[jira] [Commented] (IOTDB-4389) [MultiLeaderConsensus] Stop 1 datanode at a time for 10 minutes ，after cluster restarts, some timeseries have more data and some have less data
[
https://issues.apache.org/jira/browse/IOTDB-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603540#comment-17603540
]
Jinrui Zhang commented on IOTDB-4389:
-
Before repro of this case, I have two assumptions regarding to data lost.
# If the leader is on the DataNode which is down, the write operations from BM
fails
# WAL is not in `Sync` mode, which will lead to data lost when restarting of
DataNode
> [MultiLeaderConsensus] Stop 1 datanode at a time for 10 minutes ，after
> cluster restarts, some timeseries have more data and some have less data
> ---
>
> Key: IOTDB-4389
> URL: https://issues.apache.org/jira/browse/IOTDB-4389
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Jinrui Zhang
>Priority: Major
> Attachments: 0913_after_cluster_restart_query.out,
> 0913_bef_stop-cluster_with_autoflush_query.out, benchmark_down_datanode.conf,
> down_datanode.sh, image-2022-09-13-11-09-33-609.png, screenshot-1.png
>
>
> master_0909_bdd7ca8 ， 3C9D
> 9个datanode，间隔30分钟，执行 1次下线 1个datanode，下线时间为10分钟。
> 9个datanode均下线1次后，不再执行故障操作。
> 客户端运行完成
> 进程退出，（70小时后）执行重启前后的数据正确性验证，{color:#DE350B}*部分序列重启后，有的丢失10个点数据，有的多10个点数据。*{color}：
> !image-2022-09-13-11-09-33-609.png!
> {color:#DE350B}*下图红框多10个点数据：*{color}
> !screenshot-1.png!
> 详细查询结果见附件。
> 测试流程
> 1. ConfigNode机器
> 172.20.70.31（leader） 8核32G
> 172.20.70.32/33 4核16G
> confignode配置参数：
> MAX_HEAP_SIZE="8G"
> MAX_DIRECT_MEMORY_SIZE="4G"
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region_consensus_protocol_class=org.apache.iotdb.consensus.multileader.MultiLeaderConsensus
> schema_replication_factor=3
> data_replication_factor=3
> 2. DataNode机器
> 172.20.70.2/3/4/5/13/14/16/18/19 8核32G
> 配置参数
> MAX_HEAP_SIZE="20G"
> MAX_DIRECT_MEMORY_SIZE="6G"
> max_connection_for_internal_service=200
> wal_buffer_size_in_byte=1048576
> enable_timed_flush_seq_memtable=true
> seq_memtable_flush_interval_in_ms=360
> seq_memtable_flush_check_interval_in_ms=60
> enable_timed_flush_unseq_memtable=true
> unseq_memtable_flush_interval_in_ms=360
> unseq_memtable_flush_check_interval_in_ms=60
> max_waiting_time_when_insert_blocked=360
> query_timeout_threshold=360
> 3. benchmark配置见附件
> CLIENT_NUMBER=50
> 运行benchmark。
> 4. 运行down datanode 脚本
> cat down_datanode.sh
> #!/bin/bash
> node1="172.20.70.4"
> node2="172.20.70.5"
> node3="172.20.70.3"
> node4="172.20.70.2"
> node5="172.20.70.13"
> node6="172.20.70.14"
> node7="172.20.70.16"
> node8="172.20.70.18"
> node9="172.20.70.19"
> cluster_dir="/data/iotdb"
> cur_cluster="master_0909_bdd7ca8"
> u_name="cluster"
> function down_datanode()
> {
> t=`date '+%Y-%m-%d %H:%M:%S'`
> echo "${t}"
> node=$1
> ${cluster_dir}/${cur_cluster}/datanode/sbin/start-cli.sh -h ${node} -e "show
> cluster"
> ${cluster_dir}/${cur_cluster}/datanode/sbin/start-cli.sh -h ${node} -e "show
> regions"
> ssh ${u_name}@${node} "source
> /etc/profile;${cluster_dir}/${cur_cluster}/datanode/sbin/stop-datanode.sh"
> sleep 10m
> ssh ${u_name}@${node} "source
> /etc/profile;${cluster_dir}/${cur_cluster}/datanode/sbin/start-datanode.sh >
> /dev/null 2>&1 &"
> sleep 30m
> }
> sleep 30m
> down_datanode ${node1}
> down_datanode ${node2}
> down_datanode ${node3}
> down_datanode ${node4}
> down_datanode ${node5}
> down_datanode ${node6}
> down_datanode ${node7}
> down_datanode ${node8}
> down_datanode ${node9}
> 5. benchmark运行16.25小时
> 6.（70小时后）执行查询 正确性验证
> 停集群前执行查询
> select count(s_5) from root.** align by device；
> 清os 缓存
> 重启集群
> select count(s_5) from root.** align by device；
> 对比2个结果
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (IOTDB-3980) Unable to obtain monitoring data for a certain period of time
[ https://issues.apache.org/jira/browse/IOTDB-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 张洪胤 reopened IOTDB-3980: > Unable to obtain monitoring data for a certain period of time > - > > Key: IOTDB-3980 > URL: https://issues.apache.org/jira/browse/IOTDB-3980 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: zhenqqiang >Assignee: 张洪胤 >Priority: Major > Labels: pull-request-available > Attachments: image-2022-07-27-15-46-01-312.png > > > Unable to obtain monitoring data for a certain period of time > !image-2022-07-27-15-46-01-312.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [iotdb-client-go] HTHou commented on a diff in pull request #50: add create aligned timeseries request
HTHou commented on code in PR #50:
URL: https://github.com/apache/iotdb-client-go/pull/50#discussion_r969447043
##
test/e2e/e2e_test.go:
##
@@ -94,6 +94,34 @@ func (s *e2eTestSuite) Test_CreateTimeseries() {
assert.Equal(timeseries, "root.tsg1.dev1.status")
}
+func (s *e2eTestSuite) Test_CreateAlignedTimeseries() {
+ var (
+ prefixPath = "root.tsg1.dev1"
+ measurements = []string{"status"}
+ measurementAlias = []string{"sts"}
+ dataTypes= []client.TSDataType{
+ client.FLOAT,
+ }
Review Comment:
Better to use two measurements to show the aligned relation of timeseries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb-client-go] HTHou commented on a diff in pull request #50: add create aligned timeseries request
HTHou commented on code in PR #50:
URL: https://github.com/apache/iotdb-client-go/pull/50#discussion_r969447043
##
test/e2e/e2e_test.go:
##
@@ -94,6 +94,34 @@ func (s *e2eTestSuite) Test_CreateTimeseries() {
assert.Equal(timeseries, "root.tsg1.dev1.status")
}
+func (s *e2eTestSuite) Test_CreateAlignedTimeseries() {
+ var (
+ prefixPath = "root.tsg1.dev1"
+ measurements = []string{"status"}
+ measurementAlias = []string{"sts"}
+ dataTypes= []client.TSDataType{
+ client.FLOAT,
+ }
Review Comment:
Better to two use two measurements to show the aligned relation of
timeseries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb-client-go] HTHou commented on a diff in pull request #50: add create aligned timeseries request
HTHou commented on code in PR #50:
URL: https://github.com/apache/iotdb-client-go/pull/50#discussion_r969445165
##
test/e2e/e2e_test.go:
##
@@ -94,6 +94,37 @@ func (s *e2eTestSuite) Test_CreateTimeseries() {
assert.Equal(timeseries, "root.tsg1.dev1.status")
}
+func (s *e2eTestSuite) Test_CreateAlignedTimeseries() {
+ var (
+ prefixPath = "root.tsg1.dev1"
+ measurements = []string{"status"}
+ measurementAlias = []string{"sts"}
Review Comment:
temperature is missing...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb-client-go] Fiekers commented on a diff in pull request #50: add create aligned timeseries request
Fiekers commented on code in PR #50:
URL: https://github.com/apache/iotdb-client-go/pull/50#discussion_r969442208
##
test/e2e/e2e_test.go:
##
@@ -94,6 +94,37 @@ func (s *e2eTestSuite) Test_CreateTimeseries() {
assert.Equal(timeseries, "root.tsg1.dev1.status")
}
+func (s *e2eTestSuite) Test_CreateAlignedTimeseries() {
+ var (
+ prefixPath = "root.tsg1.dev1"
+ measurements = []string{"status"}
+ measurementAlias = []string{"sts"}
Review Comment:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb-client-go] HTHou commented on a diff in pull request #50: add create aligned timeseries request
HTHou commented on code in PR #50:
URL: https://github.com/apache/iotdb-client-go/pull/50#discussion_r969441157
##
test/e2e/e2e_test.go:
##
@@ -94,6 +94,37 @@ func (s *e2eTestSuite) Test_CreateTimeseries() {
assert.Equal(timeseries, "root.tsg1.dev1.status")
}
+func (s *e2eTestSuite) Test_CreateAlignedTimeseries() {
+ var (
+ prefixPath = "root.tsg1.dev1"
+ measurements = []string{"status"}
+ measurementAlias = []string{"sts"}
+ dataTypes= []client.TSDataType{
+ client.FLOAT,
+ client.FLOAT,
+ }
Review Comment:
I mean line 102 has two datatype, therefore there should be two measurements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb-client-go] Fiekers commented on a diff in pull request #50: add create aligned timeseries request
Fiekers commented on code in PR #50:
URL: https://github.com/apache/iotdb-client-go/pull/50#discussion_r969439270
##
test/e2e/e2e_test.go:
##
@@ -94,6 +94,37 @@ func (s *e2eTestSuite) Test_CreateTimeseries() {
assert.Equal(timeseries, "root.tsg1.dev1.status")
}
+func (s *e2eTestSuite) Test_CreateAlignedTimeseries() {
+ var (
+ prefixPath = "root.tsg1.dev1"
+ measurements = []string{"status"}
+ measurementAlias = []string{"sts"}
Review Comment:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb-client-go] Fiekers commented on a diff in pull request #50: add create aligned timeseries request
Fiekers commented on code in PR #50:
URL: https://github.com/apache/iotdb-client-go/pull/50#discussion_r969438946
##
test/e2e/e2e_test.go:
##
@@ -94,6 +94,37 @@ func (s *e2eTestSuite) Test_CreateTimeseries() {
assert.Equal(timeseries, "root.tsg1.dev1.status")
}
+func (s *e2eTestSuite) Test_CreateAlignedTimeseries() {
+ var (
+ prefixPath = "root.tsg1.dev1"
+ measurements = []string{"status"}
+ measurementAlias = []string{"sts"}
Review Comment:
This is the alias of measurement。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb-client-go] HTHou commented on a diff in pull request #50: add create aligned timeseries request
HTHou commented on code in PR #50:
URL: https://github.com/apache/iotdb-client-go/pull/50#discussion_r969429690
##
test/e2e/e2e_test.go:
##
@@ -94,6 +94,37 @@ func (s *e2eTestSuite) Test_CreateTimeseries() {
assert.Equal(timeseries, "root.tsg1.dev1.status")
}
+func (s *e2eTestSuite) Test_CreateAlignedTimeseries() {
+ var (
+ prefixPath = "root.tsg1.dev1"
+ measurements = []string{"status"}
+ measurementAlias = []string{"sts"}
Review Comment:
set two measurements to match the number of dataTypes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (IOTDB-4188) Optimize write concurrency and improve write throughput
[ https://issues.apache.org/jira/browse/IOTDB-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603476#comment-17603476 ] fuliwen commented on IOTDB-4188: https://github.com/apache/iotdb/pull/7281 > Optimize write concurrency and improve write throughput > --- > > Key: IOTDB-4188 > URL: https://issues.apache.org/jira/browse/IOTDB-4188 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: fuliwen >Assignee: fuliwen >Priority: Major > > Insert and query, compaction, and so on share a read-write lock, affecting > write performance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4364) Reduce read amplication in compaction
[ https://issues.apache.org/jira/browse/IOTDB-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603471#comment-17603471 ] 周沛辰 commented on IOTDB-4364: When compacting, it is necessary to obtain devices and timeseries from all source files, as well as the metadata of each timeseries. To avoid traversing the Zesong tree multiple times, we record the intermediate nodes of the Zesong tree. The comparison results are below： !image-2022-09-13-17-00-39-214.png|width=907,height=500! !image-2022-09-13-17-01-06-224.png|width=695,height=400! > Reduce read amplication in compaction > - > > Key: IOTDB-4364 > URL: https://issues.apache.org/jira/browse/IOTDB-4364 > Project: Apache IoTDB > Issue Type: Bug >Reporter: 周沛辰 >Assignee: 周沛辰 >Priority: Major > Labels: pull-request-available > Attachments: image-2022-09-08-11-45-36-035.png, > image-2022-09-13-17-00-39-214.png, image-2022-09-13-17-01-06-224.png > > > In massive timeseries scenarios, each device has 20 timeseries, and each file > has tens of thousands of devices, that is, millions of timeseries. Read > throughput in cross compaction is 100 times greater than write throughput. > > !image-2022-09-08-11-45-36-035.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4396) Refactor AbstractRetryHandler in ConfigNode
Jinrui Zhang created IOTDB-4396: --- Summary: Refactor AbstractRetryHandler in ConfigNode Key: IOTDB-4396 URL: https://issues.apache.org/jira/browse/IOTDB-4396 Project: Apache IoTDB Issue Type: Bug Reporter: Jinrui Zhang Assignee: Gaofei Cao Current implementation of AbstractRetryHandler is confusing. The aim of it is to simplify the implementation of RETRY when sending request to several datanodes. But it doesn't encapsulate the logic inside the abstract class but let the implementations to maintain the retry status, which is un-safe. Refactor target: # Encapsulate the `countDownLatch` inside the abstract class rather than make it as a constructor variable # Maintain the remove logic of DataNodeLocations insides the class -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-4380) The log dispatcher is inconsistent with the search index of the wal node
[
https://issues.apache.org/jira/browse/IOTDB-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haiming Zhu reassigned IOTDB-4380:
--
Assignee: Haiming Zhu (was: 张洪胤)
> The log dispatcher is inconsistent with the search index of the wal node
>
>
> Key: IOTDB-4380
> URL: https://issues.apache.org/jira/browse/IOTDB-4380
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Haiming Zhu
>Priority: Major
> Attachments: ip74_logs.tar.gz, more_metadata.conf
>
>
> m_0908_7915b3f。
> 问题描述
> {color:#DE350B}*log dispatcher与wal node的search index不一致 ，
> datanode重启失败*{color}：
> 2022-09-09 16:32:00,011 [pool-33-IoTDB-LogDispatcher-DataRegion[12]-2] INFO
> o.a.i.d.w.n.WALNode$PlanNodeIterator:695 - timeout when waiting for next WAL
> entry ready, execute rollWALFile. {color:#DE350B}*Current search index in wal
> buffer is 2959, and next target index is 2501 *{color}
> MultiLeaderConsensus，3副本3节点
> 1. 创建元数据过程中，kill ip74 的datanode PID
> benchmark配置文件见附件。
> 2. 清空ip74 的操作系统缓存，启动ip74的datanode
> 3. 再次重新运行benchmark同一配置，IS_DELETE_DATA=true
> 这个参数为true，会先执行delete storage group root.test.*;
> benchmark运行完成，stop ip74的datanode服务
> 备份data 为/data/mpp_test/m_0908_7915b3f/datanode/data_for_recovery_Test
> 4. 清ip74操作系统缓存，启动datanode服务
> 再次运行benchmark同一配置，benchmark运行完成，
> 查看ip74的日志，看到
> 2022-09-09 15:43:13,691 [pool-23-IoTDB-MPPDataExchangeRPC-Processor-40] ERROR
> o.a.t.ProcessFunction:47 - Internal error processing getDataBlock
> org.apache.thrift.TException: Source fragment instance not found. Fragment
> instance ID: TFragmentInstanceId(queryId:20220909_074205_19400_3,
> fragmentId:2, instanceId:0).
> at
> org.apache.iotdb.db.mpp.execution.exchange.MPPDataExchangeManager$MPPDataExchangeServiceImpl.getDataBlock(MPPDataExchangeManager.java:90)
> at
> org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Processor$getDataBlock.getResult(MPPDataExchangeService.java:326)
> at
> org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Processor$getDataBlock.getResult(MPPDataExchangeService.java:306)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2022-09-09 15:43:15,312 [20220909_074205_19400_3.2.0.SinkHandle-3074] ERROR
> o.a.i.d.m.e.e.SinkHandle:281 - The TsBlock doesn't exist. Sequence ID is 1,
> remaining map is
> [0=]
> 2022-09-09 15:43:17,119 [pool-23-IoTDB-MPPDataExchangeRPC-Processor-22] ERROR
> o.a.t.ProcessFunction:47 - Internal error processing getDataBlock
> java.lang.IllegalStateException: The data block doesn't exist. Sequence ID: 1
> at
> org.apache.iotdb.db.mpp.execution.exchange.SinkHandle.getSerializedTsBlock(SinkHandle.java:285)
> at
> org.apache.iotdb.db.mpp.execution.exchange.MPPDataExchangeManager$MPPDataExchangeServiceImpl.getDataBlock(MPPDataExchangeManager.java:97)
> at
> org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Processor$getDataBlock.getResult(MPPDataExchangeService.java:326)
> at
> org.apache.iotdb.mpp.rpc.thrift.MPPDataExchangeService$Processor$getDataBlock.getResult(MPPDataExchangeService.java:306)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 5. 停止ip74的datanode服务
> 备份data 到/data/mpp_test/m_0908_7915b3f/datanode/data_for_recovery_Test_2
> 清ip74操作系统缓存，启动ip74的datanode ，失败：
> 2022-09-09 16:44:00,039 [pool-33-IoTDB-LogDispatcher-DataRegion[12]-2] INFO
> o.a.i.d.w.n.WALNode$PlanNodeIterator:695 - timeout when waiting for next WAL
> entry ready, execute rollWALFile. Current search index in wal buffer is 2959,
> and next target index is 2501
> 机器 与 集群配置
> 1. 192.168.10.72/ 73/74 48核384G
> benchmark 在71
> 2. 集群参数
> confignode
> MAX_HEAP_SIZE="8G"
> schema_region_consensus_protocol_class=org.apache.iotdb.consensus.ratis.RatisConsensus
> data_region
[jira] [Created] (IOTDB-4395) [cluster]Monitoring can support any disk information
xiaozhihong created IOTDB-4395: -- Summary: [cluster]Monitoring can support any disk information Key: IOTDB-4395 URL: https://issues.apache.org/jira/browse/IOTDB-4395 Project: Apache IoTDB Issue Type: Improvement Components: Others Reporter: xiaozhihong Assignee: 张洪胤 In the current cluster version, only the information of the system disk in the root directory can be supported, and other data disks or mounted disks cannot be monitored. It is recommended to monitor the information of any disk. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4373) During monitoring, Disk-Space in the dashboard failed to obtain disk information
[ https://issues.apache.org/jira/browse/IOTDB-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603445#comment-17603445 ] xiaozhihong commented on IOTDB-4373: Currently, the system disk information in the monitoring root directory can only be seen in the cluster version. Like this: !image-2022-09-13-15-59-00-349.png|width=477,height=264!!image-2022-09-13-15-59-25-158.png|width=478,height=142! > During monitoring, Disk-Space in the dashboard failed to obtain disk > information > > > Key: IOTDB-4373 > URL: https://issues.apache.org/jira/browse/IOTDB-4373 > Project: Apache IoTDB > Issue Type: Improvement > Components: Others >Reporter: xiaozhihong >Assignee: 张洪胤 >Priority: Major > Labels: pull-request-available > Attachments: image-2022-09-09-10-24-36-035.png, > image-2022-09-13-15-59-00-349.png, image-2022-09-13-15-59-25-158.png > > > During monitoring, Disk-Space in the dashboard cannot obtain the disk usage > of the server where the IoTDB is located. > !image-2022-09-09-10-24-36-035.png|width=554,height=255! -- This message was sent by Atlassian Jira (v8.20.10#820010)
