[jira] [Commented] (IOTDB-4400) [new stand-alone]enableMetric, the write performance does not meet expectations
[ https://issues.apache.org/jira/browse/IOTDB-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688304#comment-17688304 ] xiaozhihong commented on IOTDB-4400: The number of monitoring indicators has little impact on write performance now. > [new stand-alone]enableMetric, the write performance does not meet > expectations > --- > > Key: IOTDB-4400 > URL: https://issues.apache.org/jira/browse/IOTDB-4400 > Project: Apache IoTDB > Issue Type: Improvement > Components: Others >Affects Versions: master branch, 0.14.0, 0.14.0-SNAPSHOT >Reporter: xiaozhihong >Assignee: Hongyin Zhang >Priority: Major > Attachments: config.properties, image-2022-09-14-10-27-26-819.png > > > commit 74fb350809b2f1488a90d6d7c420f27ec14b24e5 > Turn on the monitoring function, and the two frameworks perform write > performance tests for different metric levels. The final result is very > confusing. Different levels or different frameworks, the performance of > writing is not noticeable. Turn off monitoring, execute writing, and also no > obvious difference is seen, and a positioning investigation needs to be done. > Details: > https://apache-iotdb.feishu.cn/docx/QUQSdbRaaoWDjQxcEz9cQjFdnsz?from=create_suite > !image-2022-09-14-10-27-26-819.png|width=528,height=275! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5531) [Metric] Update IoT Consensus Metric Precision
Hongyin Zhang created IOTDB-5531: Summary: [Metric] Update IoT Consensus Metric Precision Key: IOTDB-5531 URL: https://issues.apache.org/jira/browse/IOTDB-5531 Project: Apache IoTDB Issue Type: Improvement Reporter: Hongyin Zhang Assignee: Hongyin Zhang Update IoT Consensus Metric Precision from ms to ns -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4400) [new stand-alone]enableMetric, the write performance does not meet expectations
[ https://issues.apache.org/jira/browse/IOTDB-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688275#comment-17688275 ] Hongyin Zhang commented on IOTDB-4400: -- ReTest: https://apache-iotdb.feishu.cn/docx/WezSdVhHFoQgkxxTevFceI2EnXo > [new stand-alone]enableMetric, the write performance does not meet > expectations > --- > > Key: IOTDB-4400 > URL: https://issues.apache.org/jira/browse/IOTDB-4400 > Project: Apache IoTDB > Issue Type: Improvement > Components: Others >Affects Versions: master branch, 0.14.0, 0.14.0-SNAPSHOT >Reporter: xiaozhihong >Assignee: Hongyin Zhang >Priority: Major > Attachments: config.properties, image-2022-09-14-10-27-26-819.png > > > commit 74fb350809b2f1488a90d6d7c420f27ec14b24e5 > Turn on the monitoring function, and the two frameworks perform write > performance tests for different metric levels. The final result is very > confusing. Different levels or different frameworks, the performance of > writing is not noticeable. Turn off monitoring, execute writing, and also no > obvious difference is seen, and a positioning investigation needs to be done. > Details: > https://apache-iotdb.feishu.cn/docx/QUQSdbRaaoWDjQxcEz9cQjFdnsz?from=create_suite > !image-2022-09-14-10-27-26-819.png|width=528,height=275! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5520) [session]If it thrown exceptions when call session.open, the test would not end with test-ng
[ https://issues.apache.org/jira/browse/IOTDB-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yukun Zhou reassigned IOTDB-5520: - Assignee: Haonan Hou (was: Yukun Zhou) > [session]If it thrown exceptions when call session.open, the test would not > end with test-ng > - > > Key: IOTDB-5520 > URL: https://issues.apache.org/jira/browse/IOTDB-5520 > Project: Apache IoTDB > Issue Type: Improvement > Components: Client/Java >Reporter: changxue >Assignee: Haonan Hou >Priority: Minor > Attachments: image-2023-02-10-18-12-32-225.png > > > If it thrown exceptions when call session.open, the test would not end with > test-ng. > I'm not sure if the same problem occurred in Junit with IDE. > !image-2023-02-10-18-12-32-225.png|width=700! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5530) [Auth] The storage dir of auth is wrong
Hongyin Zhang created IOTDB-5530: Summary: [Auth] The storage dir of auth is wrong Key: IOTDB-5530 URL: https://issues.apache.org/jira/browse/IOTDB-5530 Project: Apache IoTDB Issue Type: Bug Reporter: Hongyin Zhang Assignee: Hongyin Zhang Attachments: image-2023-02-13-20-27-25-892.png !image-2023-02-13-20-27-25-892.png|width=634,height=257! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5529) Optimize Fast Compaction performance
[ https://issues.apache.org/jira/browse/IOTDB-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17687860#comment-17687860 ] 周沛辰 commented on IOTDB-5529: 下面是改进后,使用 ReadPoint 与 Fast 的合并性能测试,Fast 的耗时为1195秒,而 ReadPoint 的耗时为1322秒。 !image-2023-02-13-17-38-59-105.png|width=2942,height=94! !image-2023-02-13-18-05-31-901.png|width=2902,height=64! > Optimize Fast Compaction performance > > > Key: IOTDB-5529 > URL: https://issues.apache.org/jira/browse/IOTDB-5529 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: 周沛辰 >Assignee: 周沛辰 >Priority: Major > Attachments: image-2023-02-13-16-58-38-922.png, > image-2023-02-13-17-38-59-105.png, image-2023-02-13-18-05-31-901.png > > Original Estimate: 48h > Remaining Estimate: 48h > > In the scenario where all seq pages overlap with unseq pages (for example, > the sequence point is 2 4 8, and the unsequence point is 1 3 5 7, etc), with > four sub-threads, Fast performance is slower than ReadPoint. > !image-2023-02-13-16-58-38-922.png|width=903,height=217! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IOTDB-4027) ERROR o.a.i.d.e.s.SnapshotLoader:94 - Exception occurs when creating links from snapshot directory to data directory
[ https://issues.apache.org/jira/browse/IOTDB-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 刘珍 reopened IOTDB-4027: --- master 0213 a916865 私有云1期 3ConfigNode : 172.16.2.23/24/25 9DataNode : 172.16.2.2/3/4/5/6/7/8/9/10 benchmark : 172.16.2.26 /data/iotdb/benchmark_v1/conf/iotdb_4027.conf benchmark启动测试,查看region信息 串行执行如下2个操作: ~停ip3的datanode ,5分钟后再启动。 ~停ip9的datanode,10分钟后再启动。 ip9的日志 /data/iotdb/m_0213_a916865/logs 有2类error: > ERROR o.a.i.d.e.s.SnapshotLoader:94 - Exception occurs when creating links > from snapshot directory to data directory > - > > Key: IOTDB-4027 > URL: https://issues.apache.org/jira/browse/IOTDB-4027 > Project: Apache IoTDB > Issue Type: Bug > Components: mpp-cluster >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Song Ziyang >Priority: Major > Labels: pull-request-available > Fix For: 0.14.0 > > Attachments: 20221229_ip8_datanode_logs.tar.gz, > image-2022-08-03-09-39-10-230.png, image-2022-08-03-09-39-48-739.png, > image-2022-09-06-17-05-21-387.png, image-2023-02-13-17-08-32-286.png, > image-2023-02-13-17-08-38-049.png, image-2023-02-13-17-08-47-147.png, > ip18_befor_stop_datanode_log.tar.gz, ip18_restart_with-error_log.tar.gz, > ip4_2000_config.properties, screenshot-1.png > > > master_0801_55b5b17 > 问题描述 > RatisConsensus,3副本3C9D,1个bm连1个datanode执行并发写入,停止1个follower节点,5分钟后启动;{color:#DE350B}*然后停止另1个follower节点10分钟后启动,此节点启动过程中报错,此节点少数据*{color}: > 2022-08-02 18:04:17,376 [pool-4-thread-1] ERROR o.a.i.d.e.s.SnapshotLoader:94 > - Exception occurs when creating links from snapshot directory to data > directory > java.io.IOException: Cannot find > /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536/sequence/root.ip4.g_0 > or > /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536/unsequence/root.ip4.g_0 > at > org.apache.iotdb.db.engine.snapshot.SnapshotLoader.createLinksFromSnapshotDirToDataDir(SnapshotLoader.java:163) > at > org.apache.iotdb.db.engine.snapshot.SnapshotLoader.loadSnapshotForStateMachine(SnapshotLoader.java:91) > at > org.apache.iotdb.db.consensus.statemachine.DataRegionStateMachine.loadSnapshot(DataRegionStateMachine.java:93) > at > org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.loadSnapshot(ApplicationStateMachineProxy.java:188) > at > org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.lambda$initialize$0(ApplicationStateMachineProxy.java:73) > at > org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:270) > at > org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.initialize(ApplicationStateMachineProxy.java:69) > at > org.apache.ratis.server.impl.ServerState.(ServerState.java:136) > at > org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:201) > at > org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$5(RaftServerProxy.java:274) > at > java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2022-08-02 18:04:17,376 [pool-4-thread-1] ERROR > o.a.i.d.c.s.DataRegionStateMachine:95 - Fail to load snapshot from > /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536 > ip18少数据,期望序列的count值是2点 > !screenshot-1.png! > 1. 复现流程 > 私有云172.20.70.2/3/4/5/13/14/16/18/19 > benchmark 在ip15(连ip4) > 停ip4/启动ip4 , 停ip18/启动ip18,ip18报错 > !image-2022-08-03-09-39-10-230.png! > !image-2022-08-03-09-39-48-739.png! > 2. 启动benchmark > 2022-08-02 17:34:57 启动bm > 3. 停止ip4的datanode > 2022-08-02 17:45:42停止datanode > sleep 300 > 启动ip4 > 4. 停止ip18的datanode > 2022-08-02 17:54:11 停止ip18的datanode > sleep 600 > 启动ip18 > {color:#DE350B}*启动过程中,报错*{color}: > 见问题描述 > bm写入完成,各节点同步完成,{color:#DE350B}*ip18节点少数据*{color},ip16,ip4 的数据正确。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5529) Optimize Fast Compaction performance
周沛辰 created IOTDB-5529: -- Summary: Optimize Fast Compaction performance Key: IOTDB-5529 URL: https://issues.apache.org/jira/browse/IOTDB-5529 Project: Apache IoTDB Issue Type: Improvement Reporter: 周沛辰 Assignee: 周沛辰 Attachments: image-2023-02-13-16-58-38-922.png In the scenario where all seq pages overlap with unseq pages (for example, the sequence point is 2 4 8, and the unsequence point is 1 3 5 7, etc), with four sub-threads, Fast performance is slower than ReadPoint. !image-2023-02-13-16-58-38-922.png|width=903,height=217! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5528) Memory statistics for each schema region
yanze chen created IOTDB-5528: - Summary: Memory statistics for each schema region Key: IOTDB-5528 URL: https://issues.apache.org/jira/browse/IOTDB-5528 Project: Apache IoTDB Issue Type: Improvement Reporter: yanze chen Assignee: yanze chen The memory statistics for the current metadata are global. For better monitoring, the granularity of memory statistics needs to be refined per SchemaRegion. -- This message was sent by Atlassian Jira (v8.20.10#820010)