[jira] [Created] (IOTDB-6010) NPE an IndexOutOfBound Exception in CPU Metrics
Liuxuxin created IOTDB-6010: --- Summary: NPE an IndexOutOfBound Exception in CPU Metrics Key: IOTDB-6010 URL: https://issues.apache.org/jira/browse/IOTDB-6010 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Attachments: image-2023-06-19-22-51-14-605.png, image-2023-06-19-22-52-06-108.png !image-2023-06-19-22-51-14-605.png! !image-2023-06-19-22-52-06-108.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5975) NullPointerException when updating Cpu metrics
Liuxuxin created IOTDB-5975: --- Summary: NullPointerException when updating Cpu metrics Key: IOTDB-5975 URL: https://issues.apache.org/jira/browse/IOTDB-5975 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Attachments: image-2023-06-07-21-03-45-246.png !image-2023-06-07-21-03-45-246.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5969) Support validate tsfile by reading them sequentially after compaction
[ https://issues.apache.org/jira/browse/IOTDB-5969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729063#comment-17729063 ] Liuxuxin commented on IOTDB-5969: - PR Links:https://github.com/apache/iotdb/pull/10049 > Support validate tsfile by reading them sequentially after compaction > - > > Key: IOTDB-5969 > URL: https://issues.apache.org/jira/browse/IOTDB-5969 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Liuxuxin >Priority: Major > > After the merge, add a process to check that the file is complete. The way to > check is to read each part of the file sequentially, which should be faster > because it's read sequentially. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5969) Support validate tsfile by reading them sequentially after compaction
Liuxuxin created IOTDB-5969: --- Summary: Support validate tsfile by reading them sequentially after compaction Key: IOTDB-5969 URL: https://issues.apache.org/jira/browse/IOTDB-5969 Project: Apache IoTDB Issue Type: New Feature Reporter: Liuxuxin After the merge, add a process to check that the file is complete. The way to check is to read each part of the file sequentially, which should be faster because it's read sequentially. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5910) Execute "drop database" when starting compaction in loop, causing memory leak in the "IoTDB-Compaction_Schedule" thread.
[ https://issues.apache.org/jira/browse/IOTDB-5910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reassigned IOTDB-5910: --- Assignee: Zhijia Cao (was: Liuxuxin) > Execute "drop database" when starting compaction in loop, causing memory leak > in the "IoTDB-Compaction_Schedule" thread. > > > Key: IOTDB-5910 > URL: https://issues.apache.org/jira/browse/IOTDB-5910 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Compaction, mpp-cluster >Affects Versions: 1.2.0 >Reporter: 刘珍 >Assignee: Zhijia Cao >Priority: Major > Attachments: image-2023-05-23-14-19-38-388.png, > image-2023-05-23-14-19-48-245.png, image-2023-05-23-14-20-02-106.png, > ip2_dn_stack_thread.out, load_insert_drop_db_1.sh, load_insert_drop_db_2.sh, > run_2_client.sh > > > 问题描述： > 合并开始执行时，drop database，循环这一过程，发现合并线程有泄漏： > !image-2023-05-23-14-19-38-388.png! > !image-2023-05-23-14-19-48-245.png! > 脚本内容如下： > !image-2023-05-23-14-20-02-106.png! > 测试环境： > 版本：iotdb master 0523_f8516ed > 机器：192.168.130.21C1D > 配置参数 > /ssd/iotdb/i_m_0523_f8516ed/conf/confignode-env.sh > MAX_HEAP_SIZE="2G" > MAX_DIRECT_MEMORY_SIZE="1G" > /ssd/iotdb/i_m_0523_f8516ed/conf/datanode-env.sh > MAX_HEAP_SIZE="16G" > MAX_DIRECT_MEMORY_SIZE="4G" > /ssd/iotdb/i_m_0523_f8516ed/conf/iotdb-common.properties > default_storage_group_level=2 > query_timeout_threshold=3600 > min_cross_compaction_unseq_file_level=0 > compaction_write_throughput_mb_per_sec=64 > /ssd/iotdb/i_m_0523_f8516ed/conf/iotdb-confignode.properties > cn_internal_address=192.168.130.2 > cn_target_config_node_list=192.168.130.2:10710 > cn_metric_reporter_list=PROMETHEUS > cn_metric_level=IMPORTANT > cn_metric_prometheus_reporter_port=9081 > /ssd/iotdb/i_m_0523_f8516ed/conf/iotdb-datanode.properties > dn_rpc_address=192.168.130.2 > dn_internal_address=192.168.130.2 > dn_target_config_node_list=192.168.130.2:10710 > dn_metric_reporter_list=PROMETHEUS > dn_metric_level=IMPORTANT > 测试脚本在/ssd/iotdb/i_m_0523_f8516ed ,包含3个脚本，运行run_2_client.sh > jstack见附件。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5897) NullPointerException In compaction
[ https://issues.apache.org/jira/browse/IOTDB-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17724988#comment-17724988 ] Liuxuxin commented on IOTDB-5897: - PR: [https://github.com/apache/iotdb/pull/9888] [https://github.com/apache/iotdb/pull/9886] > NullPointerException In compaction > -- > > Key: IOTDB-5897 > URL: https://issues.apache.org/jira/browse/IOTDB-5897 > Project: Apache IoTDB > Issue Type: Bug >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: master branch, 1.1.1 > > Attachments: image-2023-05-18-13-13-24-017.png > > > !image-2023-05-18-13-13-24-017.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5903) Cannot select any inner space compaction task when there is only unsequence data
Liuxuxin created IOTDB-5903: --- Summary: Cannot select any inner space compaction task when there is only unsequence data Key: IOTDB-5903 URL: https://issues.apache.org/jira/browse/IOTDB-5903 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Attachments: image-2023-05-19-16-16-44-423.png !image-2023-05-19-16-16-44-423.png! The reason is that the tsfile manager will only return the time partition id in sequence files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5897) NullPointerException In compaction
Liuxuxin created IOTDB-5897: --- Summary: NullPointerException In compaction Key: IOTDB-5897 URL: https://issues.apache.org/jira/browse/IOTDB-5897 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 1.1.1 Attachments: image-2023-05-18-13-13-24-017.png !image-2023-05-18-13-13-24-017.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5896) Failed to execute delete statement
Liuxuxin created IOTDB-5896: --- Summary: Failed to execute delete statement Key: IOTDB-5896 URL: https://issues.apache.org/jira/browse/IOTDB-5896 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Haonan Hou Fix For: master branch, 1.1.1 Attachments: image-2023-05-18-13-12-31-713.png !image-2023-05-18-13-12-31-713.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5889) TTL Cannot delete expired tsfiles
Liuxuxin created IOTDB-5889: --- Summary: TTL Cannot delete expired tsfiles Key: IOTDB-5889 URL: https://issues.apache.org/jira/browse/IOTDB-5889 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 1.1.1 Attachments: image-2023-05-17-11-37-59-752.png, image-2023-05-17-11-38-36-754.png !image-2023-05-17-11-37-59-752.png! IoTDB cannot delete expired tsfiles with a three days TTL. !image-2023-05-17-11-38-36-754.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5844) Compaction module seems to get stuck
Liuxuxin created IOTDB-5844: --- Summary: Compaction module seems to get stuck Key: IOTDB-5844 URL: https://issues.apache.org/jira/browse/IOTDB-5844 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 1.1.0 Attachments: image-2023-05-07-15-21-43-672.png, image-2023-05-07-15-22-01-641.png During cross-space compaction, sometimes due to the selection strategy of the selector, the system may execute compaction tasks that require a large amount of memory. The memory demand of a single compaction task may be larger than the total memory size allocated to the compaction module by the system. In this case, these compaction tasks will be stuck at the memory allocation step, waiting indefinitely. Subsequent compaction tasks cannot be executed either, making the compaction module appear as if it has "stopped". !image-2023-05-07-15-21-43-672.png! !image-2023-05-07-15-22-01-641.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5819) Network Metrics Failed to start
Liuxuxin created IOTDB-5819: --- Summary: Network Metrics Failed to start Key: IOTDB-5819 URL: https://issues.apache.org/jira/browse/IOTDB-5819 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 1.1.0 Attachments: image-2023-04-25-15-25-26-773.png Reporting NPE when booting IoTDB. !image-2023-04-25-15-25-26-773.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5815) NPE when using UDF to query
Liuxuxin created IOTDB-5815: --- Summary: NPE when using UDF to query Key: IOTDB-5815 URL: https://issues.apache.org/jira/browse/IOTDB-5815 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Fix For: master branch, 1.1.0 Attachments: image-2023-04-25-09-27-41-577.png !image-2023-04-25-09-27-41-577.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5802) Query cannot execute after memory is
Liuxuxin created IOTDB-5802: --- Summary: Query cannot execute after memory is Key: IOTDB-5802 URL: https://issues.apache.org/jira/browse/IOTDB-5802 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Fix For: 1.1.0 Attachments: image-2023-04-22-14-48-49-583.png Using 3C3D deployment, perform query simulation load and write simulation load. Through the monitoring panel, it is discovered that only one of the three nodes is executing queries. After a period of time (4 days), the node executing the queries experiences frequent FGCs, with CPU usage consistently reaching 100%, and both write and query operations become unresponsive; the other two nodes have no abnormalities, but the write traffic begins to decline. After killing the query process, observe the monitoring panel of the FGC-affected node, and about 10 minutes later, FGCs no longer occur, CPU utilization drops below 40%, the write throughput starts to rise. And at this time, any query from the query load executed through the CLI will result in the error shown in the following figure. !image-2023-04-22-14-48-49-583.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5801) IllegalStateException occurs while getting data blocks
Liuxuxin created IOTDB-5801: --- Summary: IllegalStateException occurs while getting data blocks Key: IOTDB-5801 URL: https://issues.apache.org/jira/browse/IOTDB-5801 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Fix For: master branch Attachments: image-2023-04-20-17-29-36-315.png !image-2023-04-20-17-29-36-315.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5796) PerformanceOverview monitoring consumes a significant amount of CPU resources.
Liuxuxin created IOTDB-5796: --- Summary: PerformanceOverview monitoring consumes a significant amount of CPU resources. Key: IOTDB-5796 URL: https://issues.apache.org/jira/browse/IOTDB-5796 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Attachments: 模拟环境-44-80w负载-cpu-4.17上午.html PerformanceOverview monitoring accounts for approximately 11% of CPU overhead. For comparison, the Compaction module also only accounts for around 11% of CPU overhead. The results of CPU profiling are shown in the attached flame graph. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5795) Using Bits instead of Bytes to Store Boolean Type Data
Liuxuxin created IOTDB-5795: --- Summary: Using Bits instead of Bytes to Store Boolean Type Data Key: IOTDB-5795 URL: https://issues.apache.org/jira/browse/IOTDB-5795 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Fix For: master branch Attachments: image-2023-04-19-14-33-24-841.png, image-2023-04-19-14-33-37-296.png In TsFile, Boolean type data typically takes up 1 byte (i.e. 8 bits) of storage space. This means that regardless of whether the Boolean value is true or false, they will occupy the same amount of space. However, since Boolean type only needs 1 bit to represent true or false, using bits to store Boolean type data can greatly reduce the required storage space. Using bits instead of bytes to store Boolean type data can greatly reduce the required storage space while improving data storage and transmission efficiency. !image-2023-04-19-14-33-24-841.png! !image-2023-04-19-14-33-37-296.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5778) In 3C3D, after shutting down each DataNode then restarting it one by one and , the SessionPool is unable to connect to the cluster.
Liuxuxin created IOTDB-5778: --- Summary: In 3C3D, after shutting down each DataNode then restarting it one by one and , the SessionPool is unable to connect to the cluster. Key: IOTDB-5778 URL: https://issues.apache.org/jira/browse/IOTDB-5778 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Fix For: 1.1.0 Attachments: mock.zip Deploy an IoTDB cluster using 3C3D and connect to the cluster for writing and querying via SessionPool. To simulate the process of cluster upgrading or parameter modification, shut down one DataNode at a time and then restart it. Perform this operation sequentially for all three DataNodes, ensuring that two DataNodes are always alive. After all three DataNodes have been restarted, the write and query programs will report errors, indicating that they cannot connect to the cluster. In the attachment, org.example.realTimeQuery.RealTimeQueryV6 tests writing, and org.example.realTimeQuery.RealTimeQueryByQuery tests querying. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5777) When writing data using non-root users, the permission authentication module takes too long
Liuxuxin created IOTDB-5777: --- Summary: When writing data using non-root users, the permission authentication module takes too long Key: IOTDB-5777 URL: https://issues.apache.org/jira/browse/IOTDB-5777 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Fix For: master branch, 1.1.0 Attachments: 20230414-162617.html, image-2023-04-17-11-27-41-532.png When writing data using non-root users, the time consumption of the permission authentication module is too high, accounting for about 2/3 of the total write time. The flame graph shows that the time consumption is mainly concentrated on the initialization of PartialPath. !image-2023-04-17-11-27-41-532.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5671) Fix file handler leak
Liuxuxin created IOTDB-5671: --- Summary: Fix file handler leak Key: IOTDB-5671 URL: https://issues.apache.org/jira/browse/IOTDB-5671 Project: Apache IoTDB Issue Type: Bug Components: Core/Compaction Affects Versions: master branch, 1.0.1, 1.1.0 Reporter: Liuxuxin Assignee: Liuxuxin Attachments: image-2023-03-14-15-17-58-568.png !image-2023-03-14-15-17-58-568.png! The files are deleted after compaction, but the inode cannot be actually deleted because the file handlers is occupied by the process, which leads to a waste of disk space. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5663) Support network monitor in metrics
Liuxuxin created IOTDB-5663: --- Summary: Support network monitor in metrics Key: IOTDB-5663 URL: https://issues.apache.org/jira/browse/IOTDB-5663 Project: Apache IoTDB Issue Type: New Feature Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master Supports monitoring network bandwidth, pps, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5662) BufferedUnderflowException occurs in inner space compaction
Liuxuxin created IOTDB-5662: --- Summary: BufferedUnderflowException occurs in inner space compaction Key: IOTDB-5662 URL: https://issues.apache.org/jira/browse/IOTDB-5662 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.13.4, 1.0.1, 1.1.0, master Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 1.0.1, 1.1.0-SNAPSHOT Attachments: image-2023-03-11-12-34-25-210.png !image-2023-03-11-12-34-25-210.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5517) Support disk io status monitor in metrics dashboard
[ https://issues.apache.org/jira/browse/IOTDB-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17686967#comment-17686967 ] Liuxuxin commented on IOTDB-5517: - The design doc is here: https://apache-iotdb.feishu.cn/docx/BspbdI8JJoxPQpxdP3KcPaKunfw > Support disk io status monitor in metrics dashboard > > > Key: IOTDB-5517 > URL: https://issues.apache.org/jira/browse/IOTDB-5517 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: master branch, 1.0.1 > > Original Estimate: 96h > Remaining Estimate: 96h > > Monitoring disk IO helps us to get a clearer picture of the database state, > but it is not currently supported by IoTDB. The purpose of this issue is to > monitor disk IO under Linux. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5517) Support disk io status monitor in metrics dashboard
Liuxuxin created IOTDB-5517: --- Summary: Support disk io status monitor in metrics dashboard Key: IOTDB-5517 URL: https://issues.apache.org/jira/browse/IOTDB-5517 Project: Apache IoTDB Issue Type: New Feature Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 1.0.1 Monitoring disk IO helps us to get a clearer picture of the database state, but it is not currently supported by IoTDB. The purpose of this issue is to monitor disk IO under Linux. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IOTDB-5504) IllegalArgumentException occurs when scheduling compaction
[ https://issues.apache.org/jira/browse/IOTDB-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reopened IOTDB-5504: - > IllegalArgumentException occurs when scheduling compaction > -- > > Key: IOTDB-5504 > URL: https://issues.apache.org/jira/browse/IOTDB-5504 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: master branch, 0.13.4, 1.0.1 >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Labels: pull-request-available > Attachments: image-2023-02-09-09-32-01-814.png > > > !image-2023-02-09-09-32-01-814.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5504) IllegalArgumentException occurs when scheduling compaction
Liuxuxin created IOTDB-5504: --- Summary: IllegalArgumentException occurs when scheduling compaction Key: IOTDB-5504 URL: https://issues.apache.org/jira/browse/IOTDB-5504 Project: Apache IoTDB Issue Type: Bug Affects Versions: master branch, 0.13.4, 1.0.1 Reporter: Liuxuxin Assignee: Liuxuxin Attachments: image-2023-02-09-09-32-01-814.png !image-2023-02-09-09-32-01-814.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5475) IllegalPathException occurs when compacting with escape device id
Liuxuxin created IOTDB-5475: --- Summary: IllegalPathException occurs when compacting with escape device id Key: IOTDB-5475 URL: https://issues.apache.org/jira/browse/IOTDB-5475 Project: Apache IoTDB Issue Type: Bug Components: Core/Engine Affects Versions: master branch, 0.13.4, 1.0.1 Reporter: Liuxuxin Attachments: image-2023-02-06-14-42-02-722.png !image-2023-02-06-14-42-02-722.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5140) Add metrics for compaction deserializing pages or writing chunks
[ https://issues.apache.org/jira/browse/IOTDB-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679097#comment-17679097 ] Liuxuxin commented on IOTDB-5140: - I have done some performance test for this PR. For inner sequence space compaction, file num is 23, total file size is 4.0GB. For inner unsequence space compaction, file num is 22, total file size is 4.7GB. For cross space compaction, sequence file num is 1, unsequence file num is 1, sequence file size is 4.0GB, unsequence file size is 4.7 GB. | | master(030302c96582f)| IOTDB-5140 | |Inner-Seq| 281 s| 271 s | |Inner-Unseq| 235 s| 239 s | |Cross| 492 s| 501 s | > Add metrics for compaction deserializing pages or writing chunks > > > Key: IOTDB-5140 > URL: https://issues.apache.org/jira/browse/IOTDB-5140 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Labels: pull-request-available > Original Estimate: 24h > Remaining Estimate: 24h > > We want to trace the count of deserlializing chunk or pages during compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5163) [monitor]the file size of sequence of datanode is different from actual facts
[
https://issues.apache.org/jira/browse/IOTDB-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675942#comment-17675942
]
Liuxuxin commented on IOTDB-5163:
-
This bug cannot be reproduced now. We guessed that the issue of IOTDB-5286
might have caused the inaccurate file count, and as the ISSUE was fixed, the
bug no longer appeared.
> [monitor]the file size of sequence of datanode is different from actual facts
> -
>
> Key: IOTDB-5163
> URL: https://issues.apache.org/jira/browse/IOTDB-5163
> Project: Apache IoTDB
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: changxue
>Assignee: Liuxuxin
>Priority: Major
> Attachments: image-2022-12-09-14-24-03-964.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> [monitor]the file size of sequence of datanode is different from actual facts
> the monitor:
> !image-2022-12-09-14-24-03-964.png!
> the actual facts on disk:
> {code}
> 51M ./datanode/data/unsequence/root.test
> 51M ./datanode/data/unsequence
> 124G ./datanode/data/sequence/root.test
> 124G ./datanode/data/sequence
> 228G ./datanode/data
> 35M ./datanode/wal/root.test-3
> 66M ./datanode/wal/root.test-0
> 53M ./datanode/wal/root.test-1
> 80M ./datanode/wal/root.test-4
> 16M ./datanode/wal/root.test-2
> 248M ./datanode/wal
> 21G ./datanode/data/snapshot/root.test-3
> 21G ./datanode/data/snapshot/root.test-0
> 22G ./datanode/data/snapshot/root.test-1
> 21G ./datanode/data/snapshot/root.test-4
> 21G ./datanode/data/snapshot/root.test-2
> 104G ./datanode/data/snapshot
> {code}
> sequence:
> {code}
> [atmos@i-rh6m726k root.test]$ du -hl -d 1
> 26G ./4
> 26G ./0
> 24G ./3
> 26G ./1
> 25G ./2
> 124G .
> {code}
> the monitor show seq is about 22.7G but the disk is 124G
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-5288) [Metric] Got a wrong number of files
[ https://issues.apache.org/jira/browse/IOTDB-5288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675918#comment-17675918 ] Liuxuxin commented on IOTDB-5288: - This bug cannot be reproduced now. We guessed that the issue of IOTDB-5286 might have caused the inaccurate file count, and as the ISSUE was fixed, the bug no longer appeared. > [Metric] Got a wrong number of files > > > Key: IOTDB-5288 > URL: https://issues.apache.org/jira/browse/IOTDB-5288 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Qingxin Feng >Assignee: Liuxuxin >Priority: Minor > Labels: pull-request-available > Attachments: image-2022-12-27-08-39-27-765.png > > Original Estimate: 24h > Remaining Estimate: 24h > > commit version: version 1.0.1-SNAPSHOT (Build: 752d7cf) > Got a wrong number of files, like below picture. > !image-2022-12-27-08-39-27-765.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5156) The backup data is twice the size of the source data
[ https://issues.apache.org/jira/browse/IOTDB-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17656484#comment-17656484 ] Liuxuxin commented on IOTDB-5156: - This is more of a file system issue, and it's hard to fix from within IoTDB. > The backup data is twice the size of the source data > > > Key: IOTDB-5156 > URL: https://issues.apache.org/jira/browse/IOTDB-5156 > Project: Apache IoTDB > Issue Type: Improvement > Components: mpp-cluster >Reporter: 刘珍 >Assignee: Liuxuxin >Priority: Major > Attachments: image-2022-12-08-21-08-54-494.png > > Original Estimate: 24h > Remaining Estimate: 24h > > master > 问题描述： > 备份iotdb的data，大小是源data 的2倍。 > cp -rp m_1207_a0b2c8c_fast2/data m_1208_7f2218b_fast2/ > 备份出来的数据大小是源data的2倍： > 源snapshot 中的文件是硬链接，备份snapshot的文件会是普通文件： > !image-2022-12-08-21-08-54-494.png! > 测试流程 > 1.启动1副本1C1D（sbin/start-standalone.sh） > config , schema ,data 是ratis ，ratis ，IoT 协议 > 2. 写入数据 > 3. 正常停止datanode（会take snapshot） > 4. 备份数据 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IOTDB-5140) Add metrics for compaction deserializing pages or writing chunks
[ https://issues.apache.org/jira/browse/IOTDB-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reopened IOTDB-5140: - > Add metrics for compaction deserializing pages or writing chunks > > > Key: IOTDB-5140 > URL: https://issues.apache.org/jira/browse/IOTDB-5140 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > We want to trace the count of deserlializing chunk or pages during compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-5319) The write speed in atoms testing declined after merging commit 5126711d
[ https://issues.apache.org/jira/browse/IOTDB-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655236#comment-17655236 ] Liuxuxin commented on IOTDB-5319: - The documnet for the troubleshooting process: https://apache-iotdb.feishu.cn/docx/V9xbdSGVMoa9VFxyQvAcq5bXn0d > The write speed in atoms testing declined after merging commit 5126711d > --- > > Key: IOTDB-5319 > URL: https://issues.apache.org/jira/browse/IOTDB-5319 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jinrui Zhang >Assignee: Liuxuxin >Priority: Major > Attachments: image-2022-12-29-18-37-25-693.png, > image-2022-12-29-18-38-45-394.png > > Original Estimate: 72h > Remaining Estimate: 72h > > !image-2022-12-29-18-37-25-693.png|width=701,height=156! > > After merging this commit, the write speed in atoms testing declined. > > We inferred that this change lead to compaction grabs more CPU/IO resources, > which decrease available resources of write/read. > > After we change the parameter `iops_per_min` from 50 to 30, the scenario > still exists. > See this snapshot, > !image-2022-12-29-18-38-45-394.png|width=541,height=346! > > Let's investigate the details. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5209) Limit the read rate of compaction
Liuxuxin created IOTDB-5209: --- Summary: Limit the read rate of compaction Key: IOTDB-5209 URL: https://issues.apache.org/jira/browse/IOTDB-5209 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 1.0.1 Currently we only limit the write speed of compaction, but the read rate is not limited. This may have a bad effect on the system performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5140) Add metrics for compaction deserializing pages or writing chunks
Liuxuxin created IOTDB-5140: --- Summary: Add metrics for compaction deserializing pages or writing chunks Key: IOTDB-5140 URL: https://issues.apache.org/jira/browse/IOTDB-5140 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Assignee: Liuxuxin We want to trace the count of deserlializing chunk or pages during compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-4595) It is recommended to add real-time monitoring data for tsfile merge
[ https://issues.apache.org/jira/browse/IOTDB-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reassigned IOTDB-4595: --- Assignee: Liuxuxin (was: 张洪胤) > It is recommended to add real-time monitoring data for tsfile merge > --- > > Key: IOTDB-4595 > URL: https://issues.apache.org/jira/browse/IOTDB-4595 > Project: Apache IoTDB > Issue Type: Improvement > Components: Connectors/Grafana >Affects Versions: 0.13.3 >Reporter: xiaozhihong >Assignee: Liuxuxin >Priority: Minor > Attachments: Apache IoTDB Dashboard v0.13.1.json, > image-2022-10-10-15-50-49-400.png > > > Currently, only sequence tsfile, unsequence tsfile, and wal folder are > monitored in the grafana monitoring interface, but file used by compaction > are not monitored. > !image-2022-10-10-15-50-49-400.png|width=531,height=121! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5092) Check if seuqence tsfiles is overlapped with each other after compaction
Liuxuxin created IOTDB-5092: --- Summary: Check if seuqence tsfiles is overlapped with each other after compaction Key: IOTDB-5092 URL: https://issues.apache.org/jira/browse/IOTDB-5092 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 1.0.0 In case of sequence tsfiles overlapped with each other, we should add a check after compaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-5004) [metrics] The seq file size in grafana is inconsistent with the actual query
[ https://issues.apache.org/jira/browse/IOTDB-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reassigned IOTDB-5004: --- Assignee: Liuxuxin (was: 张洪胤) > [metrics] The seq file size in grafana is inconsistent with the actual query > > > Key: IOTDB-5004 > URL: https://issues.apache.org/jira/browse/IOTDB-5004 > Project: Apache IoTDB > Issue Type: Bug > Components: Connectors/Grafana, mpp-cluster >Affects Versions: 0.14.0-SNAPSHOT >Reporter: xiaozhihong >Assignee: Liuxuxin >Priority: Major > Attachments: image-2022-11-21-18-18-53-090.png, > image-2022-11-21-18-19-28-703.png > > > Start 3C3D, 1 replication, executed writing for 3 hours, and check that the > size file of seq is incorrect. > !image-2022-11-21-18-19-28-703.png! > !image-2022-11-21-18-18-53-090.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-5029) Change merge statement to compact, and add debug info for compaction when executing this statement
Liuxuxin created IOTDB-5029: --- Summary: Change merge statement to compact, and add debug info for compaction when executing this statement Key: IOTDB-5029 URL: https://issues.apache.org/jira/browse/IOTDB-5029 Project: Apache IoTDB Issue Type: Improvement Components: Core/Server Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch In new cluster, the statement to trigger compaction in all datanodes is named `merge`, which is not proper. It's better to change it as `compact` or `trigger compaction`. What's more, it can be helpful to print some debug log of compaction selection when executing this statement. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4939) Remove unsupport compress type
Liuxuxin created IOTDB-4939: --- Summary: Remove unsupport compress type Key: IOTDB-4939 URL: https://issues.apache.org/jira/browse/IOTDB-4939 Project: Apache IoTDB Issue Type: Task Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch Some compress methods are not supported now, remove them from code and docs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-3633) [ compaction & raft snapshot ] When recovery , the merged files without snapshot are restored to level-0 tsfiles
[ https://issues.apache.org/jira/browse/IOTDB-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627558#comment-17627558 ] Liuxuxin commented on IOTDB-3633: - One possible way to fix this problem would be to trigger a snapshot after a compaction ends and notify the consensus that a new snapshot has been created. However, this allows the consensus layer to couple to the compaction module, and it requires additional logic to handle as a standalone server without the consensus layer. > [ compaction & raft snapshot ] When recovery , the merged files without > snapshot are restored to level-0 tsfiles > > > Key: IOTDB-3633 > URL: https://issues.apache.org/jira/browse/IOTDB-3633 > Project: Apache IoTDB > Issue Type: Improvement > Components: mpp-cluster >Affects Versions: 0.14.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Liuxuxin >Priority: Major > Attachments: image-2022-06-23-16-55-05-097.png, > image-2022-06-23-16-56-05-675.png > > > 1. 停止datanode（kill -9），没有做snapshot，有合并文件 > !image-2022-06-23-16-55-05-097.png! > 2. 重启恢复，恢复后的合并文件消解成合并发生前的0层文件 > !image-2022-06-23-16-56-05-675.png! > 期望，raft log恢复后，已经合并的tsfile，还是合并状态。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4834) Remove data in compaction when the datatype of one series is not consistent in different tsfile
Liuxuxin created IOTDB-4834: --- Summary: Remove data in compaction when the datatype of one series is not consistent in different tsfile Key: IOTDB-4834 URL: https://issues.apache.org/jira/browse/IOTDB-4834 Project: Apache IoTDB Issue Type: New Feature Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.4-SNAPSHOT In some scenario, the data type of one series is different in different tsfile for some reason(eg, the bug in auto data type inference). Compaction should remove the data whose type is different with the type in MManager. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4791) readChunkMetadataAndConstructIndexTree is too slow for sealing TsFile
[
https://issues.apache.org/jira/browse/IOTDB-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626475#comment-17626475
]
Liuxuxin commented on IOTDB-4791:
-
Test program for TsFileIOWriter::endFile
public class TsFileIOWriterEndFileTest {
public static void main(String[] args) throws Exception {
TsFileIOWriter writer = new TsFileIOWriter(new File("test.tsfile"));
for (int deviceIndex = 0; deviceIndex < 1000; deviceIndex++) {
writer.startChunkGroup("root.sg.d" + deviceIndex);
for (int seriesIndex = 0; seriesIndex < 1000; seriesIndex++) {
ChunkWriterImpl chunkWriter =
new ChunkWriterImpl(
new MeasurementSchema(
"s" + seriesIndex, TSDataType.INT32, TSEncoding.RLE, CompressionType.GZIP));
for (long time = 0; time < 10; ++time) {
chunkWriter.write(time, 0);
}
chunkWriter.writeToFileWriter(writer);
}
writer.endChunkGroup();
}
writer.endFile();
}
}
> readChunkMetadataAndConstructIndexTree is too slow for sealing TsFile
> -
>
> Key: IOTDB-4791
> URL: https://issues.apache.org/jira/browse/IOTDB-4791
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/TsFile
>Reporter: yusicheng
>Assignee: Liuxuxin
>Priority: Major
> Attachments: image-2022-10-28-16-48-57-842.png
>
>
> !image-2022-10-28-16-48-57-842.png!
> you can see this method spend almost half time on Path:, is there any
> possible to speed up this method?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-4791) readChunkMetadataAndConstructIndexTree is too slow for sealing TsFile
[
https://issues.apache.org/jira/browse/IOTDB-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626474#comment-17626474
]
Liuxuxin commented on IOTDB-4791:
-
For TsFileIOWriter::endFile, I also run a test for both 0.13 and master. The
test program writes 1 millions chunk into a tsfile, then calls the endFile
method and records the time cost of this function.
*CPU: Ryzen 5800H*
*OS: Windows 10 Home*
*Memory: 16 GB*
*Java Version: Oracle JDK 1.8.0_291*
{*}Disk: WD SN730 500G{*}{*}{*}
master(3648dbea7acadd54c3fa9ce06fe): 4938 ms
rel/0.13(6147c6d5cdafdadc5bc6b08b4aa): 1311 ms
*The performance gap between them is 4x roughly.*
> readChunkMetadataAndConstructIndexTree is too slow for sealing TsFile
> -
>
> Key: IOTDB-4791
> URL: https://issues.apache.org/jira/browse/IOTDB-4791
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/TsFile
>Reporter: yusicheng
>Assignee: Liuxuxin
>Priority: Major
> Attachments: image-2022-10-28-16-48-57-842.png
>
>
> !image-2022-10-28-16-48-57-842.png!
> you can see this method spend almost half time on Path:, is there any
> possible to speed up this method?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-4791) readChunkMetadataAndConstructIndexTree is too slow for sealing TsFile
[
https://issues.apache.org/jira/browse/IOTDB-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626457#comment-17626457
]
Liuxuxin commented on IOTDB-4791:
-
Test program:
public static String getRandomString(int length) {
String str = "abcdefghijklmnopqrstuvwxyzABCDEGHIJKLMNOPQRSUVWXYZ";
Random random = new Random();
StringBuffer sb = new StringBuffer();
int len = str.length();
for (int i = 0; i < length; i++) {
int number = random.nextInt(len);
sb.append(str.charAt(number));
}
return sb.toString();
}
public static void main(String[] args) {
long totalCost = 0;
for (int i = 0; i < 100; ++i) {
StringBuilder builder = new StringBuilder().append("root.");
for (int j = 0; j < 5; ++j) {
builder.append(getRandomString(5));
builder.append(".");
}
builder.append(getRandomString(5));
long startTime = System.nanoTime();
new Path(builder.toString(), true);
totalCost += System.nanoTime() - startTime;
}
System.out.println(totalCost);
}
> readChunkMetadataAndConstructIndexTree is too slow for sealing TsFile
> -
>
> Key: IOTDB-4791
> URL: https://issues.apache.org/jira/browse/IOTDB-4791
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/TsFile
>Reporter: yusicheng
>Assignee: Liuxuxin
>Priority: Major
> Attachments: image-2022-10-28-16-48-57-842.png
>
>
> !image-2022-10-28-16-48-57-842.png!
> you can see this method spend almost half time on Path:, is there any
> possible to speed up this method?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-4791) readChunkMetadataAndConstructIndexTree is too slow for sealing TsFile
[ https://issues.apache.org/jira/browse/IOTDB-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626450#comment-17626450 ] Liuxuxin commented on IOTDB-4791: - Since the parsing method of path is different in 0.13 and master, I test the performance of the constructor of `Path` in the two version. For each version, the test program generates 1 million string, the length of each string is 40. *CPU: Ryzen 5800H* *OS: Windows 10 Home* *Memory: 16 GB* *Java Version: Oracle JDK 1.8.0_291* master(3648dbea7acadd54c3fa9ce06fe): 4.88 s rel/0.13(6147c6d5cdafdadc5bc6b08b4aa): 0.19s The performance gap between them is 25x roughly. > readChunkMetadataAndConstructIndexTree is too slow for sealing TsFile > - > > Key: IOTDB-4791 > URL: https://issues.apache.org/jira/browse/IOTDB-4791 > Project: Apache IoTDB > Issue Type: Improvement > Components: Core/TsFile >Reporter: yusicheng >Assignee: Liuxuxin >Priority: Major > Attachments: image-2022-10-28-16-48-57-842.png > > > !image-2022-10-28-16-48-57-842.png! > you can see this method spend almost half time on Path:, is there any > possible to speed up this method? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4780) Support rewrite not-aligned series as aligned series
Liuxuxin created IOTDB-4780: --- Summary: Support rewrite not-aligned series as aligned series Key: IOTDB-4780 URL: https://issues.apache.org/jira/browse/IOTDB-4780 Project: Apache IoTDB Issue Type: New Feature Components: Tools/Others Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch If the data in a existing tsfile is not aligned, and we want to rewrite to a new iotdb as aligned series to get performance improvement. Current rewrite tool does not support this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4693) Support broken tsfile rewrite
Liuxuxin created IOTDB-4693: --- Summary: Support broken tsfile rewrite Key: IOTDB-4693 URL: https://issues.apache.org/jira/browse/IOTDB-4693 Project: Apache IoTDB Issue Type: New Feature Components: TsFile Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch TsFileRewriteTool only supports rewriting complete tsfile to IoTDB. If we want to load good chunks from a broken TsFile, RewriteTool does not support this feature. This feature can be used to restore as much data as possible from TsFiles that have been corrupted by accident (write errors, disk corruption, etc.). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4650) Support read metadata then read chunk in RewriteTsFileTool
Liuxuxin created IOTDB-4650: --- Summary: Support read metadata then read chunk in RewriteTsFileTool Key: IOTDB-4650 URL: https://issues.apache.org/jira/browse/IOTDB-4650 Project: Apache IoTDB Issue Type: New Feature Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.3 Current RewriteTsFileTool read the broken tsfile from head to tail, but in some case we want the tool to read metadata first, then read the chunk according to metadata. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4636) IndexOutOfBoundsException when compacting aligned series
Liuxuxin created IOTDB-4636: --- Summary: IndexOutOfBoundsException when compacting aligned series Key: IOTDB-4636 URL: https://issues.apache.org/jira/browse/IOTDB-4636 Project: Apache IoTDB Issue Type: Bug Affects Versions: master branch, 0.13.3 Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.3 Attachments: image-2022-10-13-15-45-07-212.png !image-2022-10-13-15-45-07-212.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4635) Data cannot be queried when flushing a memtable
Liuxuxin created IOTDB-4635: --- Summary: Data cannot be queried when flushing a memtable Key: IOTDB-4635 URL: https://issues.apache.org/jira/browse/IOTDB-4635 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.13.3 Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.13.3 The data in a memtable cannot be queried when it is flushing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4579) The start time and end time of cross space compaction target file is not right
Liuxuxin created IOTDB-4579: --- Summary: The start time and end time of cross space compaction target file is not right Key: IOTDB-4579 URL: https://issues.apache.org/jira/browse/IOTDB-4579 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.13.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4518) There is .cmt file remain in disk
Liuxuxin created IOTDB-4518: --- Summary: There is .cmt file remain in disk Key: IOTDB-4518 URL: https://issues.apache.org/jira/browse/IOTDB-4518 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch Attachments: image-2022-09-23-17-59-11-345.png !image-2022-09-23-17-59-11-345.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4451) Print log when a tsfile is compacted too many times in cross space compaction
Liuxuxin created IOTDB-4451: --- Summary: Print log when a tsfile is compacted too many times in cross space compaction Key: IOTDB-4451 URL: https://issues.apache.org/jira/browse/IOTDB-4451 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.3 If a tsfile is compacted too many times, it hurts a lot to the performance. We should print log to show why this tsfile is chosen to be compacted, in order that we can figure out the reason and avoid it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4257) Design and implement ReadChunkCompactionMemoryEstimator
[ https://issues.apache.org/jira/browse/IOTDB-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603831#comment-17603831 ] Liuxuxin commented on IOTDB-4257: - Design doc: https://apache-iotdb.feishu.cn/docx/doxcnwoL4ROgQ37l2yIf6r99ffh > Design and implement ReadChunkCompactionMemoryEstimator > --- > > Key: IOTDB-4257 > URL: https://issues.apache.org/jira/browse/IOTDB-4257 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: master branch, 0.13.2 > > > Currently there is memory estimator for cross space compaction, and a memory > control framework for compaction. We need to design and implement -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-3455) Make data_region_num take effect in new standalone
[ https://issues.apache.org/jira/browse/IOTDB-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603830#comment-17603830 ] Liuxuxin commented on IOTDB-3455: - Writing with 1-vsg !image-2022-09-14-09-04-41-108.png! Writing with 3 vsg: !image-2022-09-14-09-04-53-244.png! > Make data_region_num take effect in new standalone > -- > > Key: IOTDB-3455 > URL: https://issues.apache.org/jira/browse/IOTDB-3455 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Assignee: Liuxuxin >Priority: Major > Labels: pull-request-available > Attachments: image-2022-09-14-09-04-30-833.png, > image-2022-09-14-09-04-41-108.png, image-2022-09-14-09-04-53-244.png > > > ConfigNode is a process used in the new cluster. To make the processing logic > clear, we need to fake a ConfigNode in the new standalone version, called > LocalConfigNode, which is a module but not a process. > > Currently, in the new Standalone version, we just allocate one DataRegion and > one SchemaRegion for each Storage Group. This means the data_region_num does > not take effect. > > We need to let data_region_num take effect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4251) Persist ChunkMetadata in TsFileIOWriter ahead of time to save memory
[ https://issues.apache.org/jira/browse/IOTDB-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602079#comment-17602079 ] Liuxuxin commented on IOTDB-4251: - Design doc: https://apache-iotdb.feishu.cn/docx/doxcn8f9DvHJEusQcktKBblTRTe > Persist ChunkMetadata in TsFileIOWriter ahead of time to save memory > > > Key: IOTDB-4251 > URL: https://issues.apache.org/jira/browse/IOTDB-4251 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: master branch, 0.13.2 > > Attachments: image.png > > > In the scenario of massive active sequences, the ChunkMetadata in > TsFileIOWriter occupies a large part of the memory, which may lead to OOM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IOTDB-4027) ERROR o.a.i.d.e.s.SnapshotLoader:94 - Exception occurs when creating links from snapshot directory to data directory
[
https://issues.apache.org/jira/browse/IOTDB-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liuxuxin reassigned IOTDB-4027:
---
Assignee: Song Ziyang (was: Liuxuxin)
> ERROR o.a.i.d.e.s.SnapshotLoader:94 - Exception occurs when creating links
> from snapshot directory to data directory
> -
>
> Key: IOTDB-4027
> URL: https://issues.apache.org/jira/browse/IOTDB-4027
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Song Ziyang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.14.0
>
> Attachments: image-2022-08-03-09-39-10-230.png,
> image-2022-08-03-09-39-48-739.png, image-2022-09-06-17-05-21-387.png,
> ip18_befor_stop_datanode_log.tar.gz, ip18_restart_with-error_log.tar.gz,
> ip4_2000_config.properties, screenshot-1.png
>
>
> master_0801_55b5b17
> 问题描述
> RatisConsensus，3副本3C9D，1个bm连1个datanode执行并发写入，停止1个follower节点，5分钟后启动；{color:#DE350B}*然后停止另1个follower节点10分钟后启动，此节点启动过程中报错，此节点少数据*{color}：
> 2022-08-02 18:04:17,376 [pool-4-thread-1] ERROR o.a.i.d.e.s.SnapshotLoader:94
> - Exception occurs when creating links from snapshot directory to data
> directory
> java.io.IOException: Cannot find
> /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536/sequence/root.ip4.g_0
> or
> /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536/unsequence/root.ip4.g_0
> at
> org.apache.iotdb.db.engine.snapshot.SnapshotLoader.createLinksFromSnapshotDirToDataDir(SnapshotLoader.java:163)
> at
> org.apache.iotdb.db.engine.snapshot.SnapshotLoader.loadSnapshotForStateMachine(SnapshotLoader.java:91)
> at
> org.apache.iotdb.db.consensus.statemachine.DataRegionStateMachine.loadSnapshot(DataRegionStateMachine.java:93)
> at
> org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.loadSnapshot(ApplicationStateMachineProxy.java:188)
> at
> org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.lambda$initialize$0(ApplicationStateMachineProxy.java:73)
> at
> org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:270)
> at
> org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.initialize(ApplicationStateMachineProxy.java:69)
> at
> org.apache.ratis.server.impl.ServerState.(ServerState.java:136)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:201)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$5(RaftServerProxy.java:274)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2022-08-02 18:04:17,376 [pool-4-thread-1] ERROR
> o.a.i.d.c.s.DataRegionStateMachine:95 - Fail to load snapshot from
> /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536
> ip18少数据，期望序列的count值是2点
> !screenshot-1.png!
> 1. 复现流程
> 私有云172.20.70.2/3/4/5/13/14/16/18/19
> benchmark 在ip15（连ip4）
> 停ip4/启动ip4 , 停ip18/启动ip18，ip18报错
> !image-2022-08-03-09-39-10-230.png!
> !image-2022-08-03-09-39-48-739.png!
> 2. 启动benchmark
> 2022-08-02 17:34:57 启动bm
> 3. 停止ip4的datanode
> 2022-08-02 17:45:42停止datanode
> sleep 300
> 启动ip4
> 4. 停止ip18的datanode
> 2022-08-02 17:54:11 停止ip18的datanode
> sleep 600
> 启动ip18
> {color:#DE350B}*启动过程中，报错*{color}：
> 见问题描述
> bm写入完成，各节点同步完成，{color:#DE350B}*ip18节点少数据*{color}，ip16，ip4 的数据正确。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-3455) Make data_region_num take effect in new standalone
[ https://issues.apache.org/jira/browse/IOTDB-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600540#comment-17600540 ] Liuxuxin commented on IOTDB-3455: - Test result should be finished before 2022-9-11 > Make data_region_num take effect in new standalone > -- > > Key: IOTDB-3455 > URL: https://issues.apache.org/jira/browse/IOTDB-3455 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Assignee: Liuxuxin >Priority: Major > Labels: pull-request-available > > ConfigNode is a process used in the new cluster. To make the processing logic > clear, we need to fake a ConfigNode in the new standalone version, called > LocalConfigNode, which is a module but not a process. > > Currently, in the new Standalone version, we just allocate one DataRegion and > one SchemaRegion for each Storage Group. This means the data_region_num does > not take effect. > > We need to let data_region_num take effect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4251) Persist ChunkMetadata in TsFileIOWriter ahead of time to save memory
[ https://issues.apache.org/jira/browse/IOTDB-4251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600539#comment-17600539 ] Liuxuxin commented on IOTDB-4251: - Codes should be finished before 2022-9-11 > Persist ChunkMetadata in TsFileIOWriter ahead of time to save memory > > > Key: IOTDB-4251 > URL: https://issues.apache.org/jira/browse/IOTDB-4251 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: master branch, 0.13.2 > > Attachments: image.png > > > In the scenario of massive active sequences, the ChunkMetadata in > TsFileIOWriter occupies a large part of the memory, which may lead to OOM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4257) Design and implement ReadChunkCompactionMemoryEstimator
[ https://issues.apache.org/jira/browse/IOTDB-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600538#comment-17600538 ] Liuxuxin commented on IOTDB-4257: - Document should be finished before 2022-9-11 > Design and implement ReadChunkCompactionMemoryEstimator > --- > > Key: IOTDB-4257 > URL: https://issues.apache.org/jira/browse/IOTDB-4257 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: master branch, 0.13.2 > > > Currently there is memory estimator for cross space compaction, and a memory > control framework for compaction. We need to design and implement -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4337) Fix failed to query data after loading snapshot from other node
Liuxuxin created IOTDB-4337: --- Summary: Fix failed to query data after loading snapshot from other node Key: IOTDB-4337 URL: https://issues.apache.org/jira/browse/IOTDB-4337 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch Attachments: image-2022-09-05-19-21-21-367.png !image-2022-09-05-19-21-21-367.png! This node fails to query data after loading snapshot from other node. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4257) Design and implement ReadPointCompactionMemoryEstimator
Liuxuxin created IOTDB-4257: --- Summary: Design and implement ReadPointCompactionMemoryEstimator Key: IOTDB-4257 URL: https://issues.apache.org/jira/browse/IOTDB-4257 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.2 Currently there is memory estimator for cross space compaction, and a memory control framework for compaction. We need to design and implement -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4251) Persist ChunkMetadata in TsFileIOWriter ahead of time to save memory
Liuxuxin created IOTDB-4251: --- Summary: Persist ChunkMetadata in TsFileIOWriter ahead of time to save memory Key: IOTDB-4251 URL: https://issues.apache.org/jira/browse/IOTDB-4251 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.2 Attachments: image.png In the scenario of massive active sequences, the ChunkMetadata in TsFileIOWriter occupies a large part of the memory, which may lead to OOM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4225) QueryContext occupy too much memory
[ https://issues.apache.org/jira/browse/IOTDB-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584233#comment-17584233 ] Liuxuxin commented on IOTDB-4225: - https://github.com/apache/iotdb/pull/7113 > QueryContext occupy too much memory > --- > > Key: IOTDB-4225 > URL: https://issues.apache.org/jira/browse/IOTDB-4225 > Project: Apache IoTDB > Issue Type: Bug >Affects Versions: 0.13.1 >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: master branch, 0.13.2 > > Attachments: image-2022-08-24-19-06-44-287.png > > > In the case of a large number of active sequences, a single QueryContext > instance takes up 1.6GB of memory. > !image-2022-08-24-19-06-44-287.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-4225) QueryContext occupy too much memory
Liuxuxin created IOTDB-4225: --- Summary: QueryContext occupy too much memory Key: IOTDB-4225 URL: https://issues.apache.org/jira/browse/IOTDB-4225 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.2 Attachments: image-2022-08-24-19-06-44-287.png In the case of a large number of active sequences, a single QueryContext instance takes up 1.6GB of memory. !image-2022-08-24-19-06-44-287.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-4027) ERROR o.a.i.d.e.s.SnapshotLoader:94 - Exception occurs when creating links from snapshot directory to data directory
[
https://issues.apache.org/jira/browse/IOTDB-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1755#comment-1755
]
Liuxuxin commented on IOTDB-4027:
-
The cause of this Bug may be due to atomicity issues when creating snapshots.
[~William Song] will take care of this issue.
> ERROR o.a.i.d.e.s.SnapshotLoader:94 - Exception occurs when creating links
> from snapshot directory to data directory
> -
>
> Key: IOTDB-4027
> URL: https://issues.apache.org/jira/browse/IOTDB-4027
> Project: Apache IoTDB
> Issue Type: Bug
> Components: mpp-cluster
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Liuxuxin
>Priority: Major
> Attachments: image-2022-08-03-09-39-10-230.png,
> image-2022-08-03-09-39-48-739.png, ip18_befor_stop_datanode_log.tar.gz,
> ip18_restart_with-error_log.tar.gz, ip4_2000_config.properties,
> screenshot-1.png
>
>
> master_0801_55b5b17
> 问题描述
> RatisConsensus，3副本3C9D，1个bm连1个datanode执行并发写入，停止1个follower节点，5分钟后启动；{color:#DE350B}*然后停止另1个follower节点10分钟后启动，此节点启动过程中报错，此节点少数据*{color}：
> 2022-08-02 18:04:17,376 [pool-4-thread-1] ERROR o.a.i.d.e.s.SnapshotLoader:94
> - Exception occurs when creating links from snapshot directory to data
> directory
> java.io.IOException: Cannot find
> /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536/sequence/root.ip4.g_0
> or
> /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536/unsequence/root.ip4.g_0
> at
> org.apache.iotdb.db.engine.snapshot.SnapshotLoader.createLinksFromSnapshotDirToDataDir(SnapshotLoader.java:163)
> at
> org.apache.iotdb.db.engine.snapshot.SnapshotLoader.loadSnapshotForStateMachine(SnapshotLoader.java:91)
> at
> org.apache.iotdb.db.consensus.statemachine.DataRegionStateMachine.loadSnapshot(DataRegionStateMachine.java:93)
> at
> org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.loadSnapshot(ApplicationStateMachineProxy.java:188)
> at
> org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.lambda$initialize$0(ApplicationStateMachineProxy.java:73)
> at
> org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:270)
> at
> org.apache.iotdb.consensus.ratis.ApplicationStateMachineProxy.initialize(ApplicationStateMachineProxy.java:69)
> at
> org.apache.ratis.server.impl.ServerState.(ServerState.java:136)
> at
> org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:201)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$5(RaftServerProxy.java:274)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2022-08-02 18:04:17,376 [pool-4-thread-1] ERROR
> o.a.i.d.c.s.DataRegionStateMachine:95 - Fail to load snapshot from
> /data/iotdb/master_0801_2de0dd8/datanode/./sbin/../data/consensus/data_region/47474747-4747-4747-4747-00010001/sm/1_354536
> ip18少数据，期望序列的count值是2点
> !screenshot-1.png!
> 1. 复现流程
> 私有云172.20.70.2/3/4/5/13/14/16/18/19
> benchmark 在ip15（连ip4）
> 停ip4/启动ip4 , 停ip18/启动ip18，ip18报错
> !image-2022-08-03-09-39-10-230.png!
> !image-2022-08-03-09-39-48-739.png!
> 2. 启动benchmark
> 2022-08-02 17:34:57 启动bm
> 3. 停止ip4的datanode
> 2022-08-02 17:45:42停止datanode
> sleep 300
> 启动ip4
> 4. 停止ip18的datanode
> 2022-08-02 17:54:11 停止ip18的datanode
> sleep 600
> 启动ip18
> {color:#DE350B}*启动过程中，报错*{color}：
> 见问题描述
> bm写入完成，各节点同步完成，{color:#DE350B}*ip18节点少数据*{color}，ip16，ip4 的数据正确。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IOTDB-3455) Make data_region_num take effect in new standalone
[ https://issues.apache.org/jira/browse/IOTDB-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reassigned IOTDB-3455: --- Assignee: Liuxuxin > Make data_region_num take effect in new standalone > -- > > Key: IOTDB-3455 > URL: https://issues.apache.org/jira/browse/IOTDB-3455 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Assignee: Liuxuxin >Priority: Major > > ConfigNode is a process used in the new cluster. To make the processing logic > clear, we need to fake a ConfigNode in the new standalone version, called > LocalConfigNode, which is a module but not a process. > > Currently, in the new Standalone version, we just allocate one DataRegion and > one SchemaRegion for each Storage Group. This means the data_region_num does > not take effect. > > We need to let data_region_num take effect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-3938) [Performance] Compare insert tablet between start-server.sh and start-new-server.sh
[
https://issues.apache.org/jira/browse/IOTDB-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572689#comment-17572689
]
Liuxuxin commented on IOTDB-3938:
-
新单机中执行插入时有将近一半的时间在 validate schema
!image-2022-07-29-06-49-12-855.png!
> [Performance] Compare insert tablet between start-server.sh and
> start-new-server.sh
> ---
>
> Key: IOTDB-3938
> URL: https://issues.apache.org/jira/browse/IOTDB-3938
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Server
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: FengQingxin
>Assignee: Liuxuxin
>Priority: Major
> Labels: pull-request-available
> Attachments: config.properties, image-2022-07-25-09-09-42-767.png,
> image-2022-07-26-16-13-08-321.png, image-2022-07-29-06-49-12-855.png,
> screenshot-1.png
>
>
> Produce Steps：
> 1.git clone and build iotdb-master
> 2.using start-server.sh to start iotd instance
> 3.using iotdb-benchmark to test sequence insert(get a
> result:{color:#00875a}green{color} line)
> 4.clean the test env and using start-new-server.sh to start a NewIoTDB
> instance
> 5.using iotdb-benchmark to test sequence insert(get a
> result:{color:#ff8b00}yellow{color} line)
> !image-2022-07-25-09-09-42-767.png|width=581,height=249!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-3938) [Performance] Compare insert tablet between start-server.sh and start-new-server.sh
[
https://issues.apache.org/jira/browse/IOTDB-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572690#comment-17572690
]
Liuxuxin commented on IOTDB-3938:
-
老单机中，对schema的处理时间只有约1/30
!image-2022-07-29-06-49-51-835.png!
> [Performance] Compare insert tablet between start-server.sh and
> start-new-server.sh
> ---
>
> Key: IOTDB-3938
> URL: https://issues.apache.org/jira/browse/IOTDB-3938
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Server
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: FengQingxin
>Assignee: Liuxuxin
>Priority: Major
> Labels: pull-request-available
> Attachments: config.properties, image-2022-07-25-09-09-42-767.png,
> image-2022-07-26-16-13-08-321.png, image-2022-07-29-06-49-12-855.png,
> image-2022-07-29-06-49-45-510.png, image-2022-07-29-06-49-51-835.png,
> screenshot-1.png
>
>
> Produce Steps：
> 1.git clone and build iotdb-master
> 2.using start-server.sh to start iotd instance
> 3.using iotdb-benchmark to test sequence insert(get a
> result:{color:#00875a}green{color} line)
> 4.clean the test env and using start-new-server.sh to start a NewIoTDB
> instance
> 5.using iotdb-benchmark to test sequence insert(get a
> result:{color:#ff8b00}yellow{color} line)
> !image-2022-07-25-09-09-42-767.png|width=581,height=249!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-3872) Benchmark supports inconsistent write frequencies for different workers
Liuxuxin created IOTDB-3872: --- Summary: Benchmark supports inconsistent write frequencies for different workers Key: IOTDB-3872 URL: https://issues.apache.org/jira/browse/IOTDB-3872 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: 张洪胤 Currently, data generated by benchmark is aligned for each device and timeseries, resulting in similar TsFile structures. For example, if tsfile 1 has d1 d2 d3, tsfile 2,3, and 4 will also have d1 d2 d3. However, in online systems, different devices and different timeseries may have different write frequencies. Therefore, the structure of written tsfiles is different. For example, TsFile 1 has d1, TsFile 2 has d2, and TsFile 3 has d1 and d2. Our Benchmark should be able to generate such data to test more scenarios. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-3739) NPE in MultiTsFileDeviceIterator
Liuxuxin created IOTDB-3739: --- Summary: NPE in MultiTsFileDeviceIterator Key: IOTDB-3739 URL: https://issues.apache.org/jira/browse/IOTDB-3739 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.1 Attachments: image-2022-07-04-19-01-59-308.png !image-2022-07-04-19-01-59-308.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IOTDB-3738) [0.12] ArrayIndexOutOfBoundException occurs when recovering cross space compaction
Liuxuxin created IOTDB-3738: --- Summary: [0.12] ArrayIndexOutOfBoundException occurs when recovering cross space compaction Key: IOTDB-3738 URL: https://issues.apache.org/jira/browse/IOTDB-3738 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Attachments: image-2022-07-04-16-51-02-563.png !image-2022-07-04-16-51-02-563.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-3040) [ TTL ] Expired tsfiles with a TTL of more than 10 hours are not deleted
[
https://issues.apache.org/jira/browse/IOTDB-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17562059#comment-17562059
]
Liuxuxin commented on IOTDB-3040:
-
[PR-6378|[https://github.com/apache/iotdb/pull/6378]] changes the way
compaction tasks are submitted. After than, the bug of TTL failure disappear.
> [ TTL ] Expired tsfiles with a TTL of more than 10 hours are not deleted
>
>
> Key: IOTDB-3040
> URL: https://issues.apache.org/jira/browse/IOTDB-3040
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Server
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Liuxuxin
>Priority: Major
> Labels: pull-request-available
> Attachments: config.properties, file status.png,
> iotdb-engine.properties, iotdb-env.sh, log-all-20220511.3.log.gz, log.png,
> logback.xml
>
>
> master_0427_b633df5
> 问题描述：
> 长测配置，设置TTL为24小时，
> grep ttl logs/log_all.log |tail -1
> 2022-04-29 08:57:07,404 [pool-11-IoTDB-TTL-Check-1] INFO
> o.a.i.d.e.s.DataRegion:1658 - Removed a file
> /data/iotdb_data/data/sequence/root.test.g_4/0/0/1651092583331-4021-1-0.tsfile{color:#DE350B}
> before Sat Sep 01 19:33:53 CST 2018 by ttl{color} (115392194000ms)
> ./tools/tsfileToolSet/print-tsfile-resource-files.sh
> 解析sg中第1个resource文件，查看时间戳范围，最大时间戳：
> device root.test.g_0.d_350, start time 1535749930200
> (2018-09-01T05:12:10.200+08:00[Asia/Shanghai]), {color:#DE350B}*end time
> 1535749948000 (2018-09-01T05:12:28+08:00[Asia/Shanghai])*{color}
> 有14个小时的过期文件没有被删除，磁盘空间无法释放。
> 1. 机器信息
> 私有云172.20.70.14/13 8C32G
> 数据库，bm配置文件见附件。
> 长测运行起来之后，24小时，设置TTL：
> #!/bin/bash
> #ttl默认大小24小时
> ttl=8640
> host=127.0.0.1
> port=6667
> function setTTLOfSg(){
> input_new_ttl=$1
> ./sbin/start-cli.sh -h $host -p $port -e "set ttl to root.** $input_new_ttl"
> }
> #计算操作系统当前时间和iotdb 序列max_time的差值
> function getDiffOfTime(){
># 操作系统当前值 秒
>os_now=`date +%s`
>max_time=`./sbin/start-cli.sh -h $host -p $port -e "show devices limit
> 1"|grep root|awk -F '|' '{print "./sbin/start-cli.sh -h " host " -p " port "
> -e \"select max_time(s_0) from " $2 " \" "}' host=$host port=$port|sh|grep "|
>"|awk -F '|' '{print $2}'`
>diffTime=$os_now*=1000))-max_time))
>echo $diffTime
> }
> # 2小时校正1次
> #sleep 24h
> for i in {1..4}
> do
>getDiffOfTime
>new_ttl=$((ttl+diffTime))
>echo $new_ttl
>setTTLOfSg $new_ttl
>sleep 2h
> done
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IOTDB-3040) [ TTL ] Expired tsfiles with a TTL of more than 10 hours are not deleted
[
https://issues.apache.org/jira/browse/IOTDB-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561330#comment-17561330
]
Liuxuxin commented on IOTDB-3040:
-
Next steps:
1. Delete the Deleted status from TsFileResource
2. Delete the RuntimeException in setStatus from TsFileResource
3. The TTL is changed from tryWriteLock to WriteLock
4. (Optional) In TTL mode, system can cancel compaction of expired files
> [ TTL ] Expired tsfiles with a TTL of more than 10 hours are not deleted
>
>
> Key: IOTDB-3040
> URL: https://issues.apache.org/jira/browse/IOTDB-3040
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Server
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Liuxuxin
>Priority: Major
> Labels: pull-request-available
> Attachments: config.properties, file status.png,
> iotdb-engine.properties, iotdb-env.sh, log-all-20220511.3.log.gz, log.png,
> logback.xml
>
>
> master_0427_b633df5
> 问题描述：
> 长测配置，设置TTL为24小时，
> grep ttl logs/log_all.log |tail -1
> 2022-04-29 08:57:07,404 [pool-11-IoTDB-TTL-Check-1] INFO
> o.a.i.d.e.s.DataRegion:1658 - Removed a file
> /data/iotdb_data/data/sequence/root.test.g_4/0/0/1651092583331-4021-1-0.tsfile{color:#DE350B}
> before Sat Sep 01 19:33:53 CST 2018 by ttl{color} (115392194000ms)
> ./tools/tsfileToolSet/print-tsfile-resource-files.sh
> 解析sg中第1个resource文件，查看时间戳范围，最大时间戳：
> device root.test.g_0.d_350, start time 1535749930200
> (2018-09-01T05:12:10.200+08:00[Asia/Shanghai]), {color:#DE350B}*end time
> 1535749948000 (2018-09-01T05:12:28+08:00[Asia/Shanghai])*{color}
> 有14个小时的过期文件没有被删除，磁盘空间无法释放。
> 1. 机器信息
> 私有云172.20.70.14/13 8C32G
> 数据库，bm配置文件见附件。
> 长测运行起来之后，24小时，设置TTL：
> #!/bin/bash
> #ttl默认大小24小时
> ttl=8640
> host=127.0.0.1
> port=6667
> function setTTLOfSg(){
> input_new_ttl=$1
> ./sbin/start-cli.sh -h $host -p $port -e "set ttl to root.** $input_new_ttl"
> }
> #计算操作系统当前时间和iotdb 序列max_time的差值
> function getDiffOfTime(){
># 操作系统当前值 秒
>os_now=`date +%s`
>max_time=`./sbin/start-cli.sh -h $host -p $port -e "show devices limit
> 1"|grep root|awk -F '|' '{print "./sbin/start-cli.sh -h " host " -p " port "
> -e \"select max_time(s_0) from " $2 " \" "}' host=$host port=$port|sh|grep "|
>"|awk -F '|' '{print $2}'`
>diffTime=$os_now*=1000))-max_time))
>echo $diffTime
> }
> # 2小时校正1次
> #sleep 24h
> for i in {1..4}
> do
>getDiffOfTime
>new_ttl=$((ttl+diffTime))
>echo $new_ttl
>setTTLOfSg $new_ttl
>sleep 2h
> done
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-3709) A loop occurred in TsFileResourceList, causing the query to fail and oom occurs
Liuxuxin created IOTDB-3709: --- Summary: A loop occurred in TsFileResourceList, causing the query to fail and oom occurs Key: IOTDB-3709 URL: https://issues.apache.org/jira/browse/IOTDB-3709 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Attachments: image-2022-06-30-20-19-45-770.png There is a loop in TsFileResourceList, and when query thread tries to get a copy of TsFileResourceList, it keep iterating itself and add each node in the linked list to an ArrayList, causing oom. !image-2022-06-30-20-21-09-137.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IOTDB-3292) [0.12] Chunk size overflow in level compaction
[
https://issues.apache.org/jira/browse/IOTDB-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561017#comment-17561017
]
Liuxuxin commented on IOTDB-3292:
-
{*}How to reproduce{*}: register serveral timeseries(eg. 5), set the
avg_series_point_number_threshold to a huge number(eg, 1000), set
seq_level_num to 10, then keeping writing data. The compaction will enlarge the
chunk size. After the serveral time, the chunk size will be larger than 2GB,
the bug will occurs.
> [0.12] Chunk size overflow in level compaction
> --
>
> Key: IOTDB-3292
> URL: https://issues.apache.org/jira/browse/IOTDB-3292
> Project: Apache IoTDB
> Issue Type: Bug
>Reporter: Liuxuxin
>Assignee: Liuxuxin
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.12.5-SNAPSHOT
>
> Attachments: image-2022-05-25-16-17-04-777.png
>
>
> In level compaction, the total size of two chunk to be merged is too large,
> leading to overflow of integer type.
> !image-2022-05-25-16-17-04-777.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IOTDB-3681) Replace RateLimiter with stable one
Title: Message Title Liuxuxin created an issue Apache IoTDB / IOTDB-3681 Replace RateLimiter with stable one Issue Type: Improvement Assignee: Liuxuxin Created: 28/Jun/22 06:57 Fix Versions: master branch Priority: Major Reporter: Liuxuxin IoTDB use com.google.common.util.concurrent.RateLimiter to limit the rate of compaction IO, but RateLimiter is unstable with a @Beta annotation. We should find a stable one to replace it. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sh
[jira] [Created] (IOTDB-3651) Stop compaction schedule when all compaction is disable
Liuxuxin created IOTDB-3651: --- Summary: Stop compaction schedule when all compaction is disable Key: IOTDB-3651 URL: https://issues.apache.org/jira/browse/IOTDB-3651 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.1 Even if we close all compaction, the system will start threads periodically for compaction scheduling. We should stop it. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3632) Add log for snapshot taker and loader
Liuxuxin created IOTDB-3632: --- Summary: Add log for snapshot taker and loader Key: IOTDB-3632 URL: https://issues.apache.org/jira/browse/IOTDB-3632 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch IoTDB does not print logs when taking snapshot and loading snapshot, it is not convenience for us to monitor it and debug it. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3589) DataRegion cannot recover from snapshot
Liuxuxin created IOTDB-3589: --- Summary: DataRegion cannot recover from snapshot Key: IOTDB-3589 URL: https://issues.apache.org/jira/browse/IOTDB-3589 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch Attachments: 微信图片编辑_20220621214047.jpg DataRegion cannot recover from snapshot when restarting IoTDB. It seems that the path of tsfiles is wrong. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3525) Apply Producer-Consumer pattern to compaction submission
Liuxuxin created IOTDB-3525: --- Summary: Apply Producer-Consumer pattern to compaction submission Key: IOTDB-3525 URL: https://issues.apache.org/jira/browse/IOTDB-3525 Project: Apache IoTDB Issue Type: Improvement Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch Currently we use a timing thread to submit compaction task from task queue to execution thread pool. In fact we can use a simple producer-consumer pattern do this job. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3451) Change compaction execution thread pool to fix size thread pool
Liuxuxin created IOTDB-3451: --- Summary: Change compaction execution thread pool to fix size thread pool Key: IOTDB-3451 URL: https://issues.apache.org/jira/browse/IOTDB-3451 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch, 0.13.1-SNAPSHOT Current compaction task execution thread pool and sub task execution thread pool are scheduled thread pool, but we don't use them to execute any timed task. It is better to change them to normal fix size thread pool. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3389) [rel/0.12] Not completed target file in compaction can be accessed by query
Liuxuxin created IOTDB-3389: --- Summary: [rel/0.12] Not completed target file in compaction can be accessed by query Key: IOTDB-3389 URL: https://issues.apache.org/jira/browse/IOTDB-3389 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.12.5-SNAPSHOT If the user setup IoTDB with LEVEL_COMPACTION, and stop it when compaction is executing, then restart IoTDB with NO_COMPACTION, the not completed tsfile in compaction can be accessed by queries, leading to exception in query. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3292) [0.12] Chunk size overflow in level compaction
Liuxuxin created IOTDB-3292: --- Summary: [0.12] Chunk size overflow in level compaction Key: IOTDB-3292 URL: https://issues.apache.org/jira/browse/IOTDB-3292 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.12.5-SNAPSHOT Attachments: image-2022-05-25-16-17-04-777.png In level compaction, the total size of two chunk to be merged is too large, leading to overflow of integer type. !image-2022-05-25-16-17-04-777.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3238) Cannot insert data to data node
Liuxuxin created IOTDB-3238: --- Summary: Cannot insert data to data node Key: IOTDB-3238 URL: https://issues.apache.org/jira/browse/IOTDB-3238 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: master branch Attachments: image-2022-05-19-16-05-55-348.png Cannot insert data to data node, and npe is throw. !image-2022-05-19-16-05-55-348.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3213) Apply visitor pattern to DataRegionStateMachine
Liuxuxin created IOTDB-3213: --- Summary: Apply visitor pattern to DataRegionStateMachine Key: IOTDB-3213 URL: https://issues.apache.org/jira/browse/IOTDB-3213 Project: Apache IoTDB Issue Type: New Feature Reporter: Liuxuxin Assignee: Liuxuxin Visitor pattern is used to process plan node in SchemaRegionStateMachine and DistributedPlanner, this issue aims to apply visitor pattern to DataRegionStateMachine. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3135) Parameter max_select_unseq_file_num_in_each_unseq_compaction doesn't work
Liuxuxin created IOTDB-3135: --- Summary: Parameter max_select_unseq_file_num_in_each_unseq_compaction doesn't work Key: IOTDB-3135 URL: https://issues.apache.org/jira/browse/IOTDB-3135 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.12.5 Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.12.5-SNAPSHOT In 0.12, the parameter max_select_unseq_file_num_in_each_unseq_compaction aims to limit the seleted files in one cross compaction, but it doesn't work. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-3113) Log the compaction speed in the end of compaction
Liuxuxin created IOTDB-3113: --- Summary: Log the compaction speed in the end of compaction Key: IOTDB-3113 URL: https://issues.apache.org/jira/browse/IOTDB-3113 Project: Apache IoTDB Issue Type: New Feature Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.14.0-SNAPSHOT, 0.13.1-SNAPSHOT Currently, the system only log the cost of time in compaction, which cannot accurately reflect the speed of compaction. This issue aims to show the speed of compaction in MB/s. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (IOTDB-2964) File not found when selecting files for cross space compaction
Liuxuxin created IOTDB-2964: --- Summary: File not found when selecting files for cross space compaction Key: IOTDB-2964 URL: https://issues.apache.org/jira/browse/IOTDB-2964 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.14.0 Attachments: image-2022-04-20-16-32-23-317.png !image-2022-04-20-16-32-23-317.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Assigned] (IOTDB-2931) Remove access to metadata manager during compaction
[ https://issues.apache.org/jira/browse/IOTDB-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reassigned IOTDB-2931: --- Assignee: Liuxuxin > Remove access to metadata manager during compaction > --- > > Key: IOTDB-2931 > URL: https://issues.apache.org/jira/browse/IOTDB-2931 > Project: Apache IoTDB > Issue Type: Bug >Reporter: Liuxuxin >Assignee: Liuxuxin >Priority: Major > Fix For: 0.14.0 > > > Currently, in the compaction process, the metadata information of time series > needs to be obtained from the MetadataManager. This can be a performance > bottleneck in distributed edition in the future. Therefore, this Issue aims > to remove the access to the MetadataManager and get the time series metadata > information directly from the TsFile. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IOTDB-2931) Remove access to metadata manager during compaction
Liuxuxin created IOTDB-2931: --- Summary: Remove access to metadata manager during compaction Key: IOTDB-2931 URL: https://issues.apache.org/jira/browse/IOTDB-2931 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Fix For: 0.14.0 Currently, in the compaction process, the metadata information of time series needs to be obtained from the MetadataManager. This can be a performance bottleneck in distributed edition in the future. Therefore, this Issue aims to remove the access to the MetadataManager and get the time series metadata information directly from the TsFile. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IOTDB-2922) Null pointer exception occurs in compaction
Liuxuxin created IOTDB-2922: --- Summary: Null pointer exception occurs in compaction Key: IOTDB-2922 URL: https://issues.apache.org/jira/browse/IOTDB-2922 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.14.0 Attachments: image-2022-04-14-18-52-36-274.png !image-2022-04-14-18-52-36-274.png! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (IOTDB-2811) [delete sg + compaction] compacting: false does not exist either, do nothing. Set system to read-only
[
https://issues.apache.org/jira/browse/IOTDB-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515197#comment-17515197
]
Liuxuxin commented on IOTDB-2811:
-
If user deletes the storage group when it is executing compaction, the
{{CompactionExceptionHandler}} may fail to handle exception because it find
that both source files and target files are lost. Consequently, the system will
be set to *read-only* because the {{CompactionExceptionHandler}} takes it as a
very serious situation.
The solution is to collect the {{Future}} object when the compaction task is
submitted to thread pool, and cancel the executing of compaction task before
deleting a storage group.
> [delete sg + compaction] compacting: false does not exist either, do nothing.
> Set system to read-only
> --
>
> Key: IOTDB-2811
> URL: https://issues.apache.org/jira/browse/IOTDB-2811
> Project: Apache IoTDB
> Issue Type: Bug
> Components: Core/Compaction
>Affects Versions: 0.14.0-SNAPSHOT
>Reporter: 刘珍
>Assignee: Liuxuxin
>Priority: Major
> Labels: pull-request-available
> Attachments: log_all.zip
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> rel/0.13 e10325f04e6d631799002dd76d12dd019cf72dc0
> 内部跑SQL功能测试：
> aligned_timeseries
> continuous_query
> issues
> maintenance_command
> privilege
> shell_scripts
> template
> trigger
> ttl
> udf
> 合并线程Set system to read-only（整个日志见附件） :
> 2022-03-25 17:01:47,597 [pool-7-IoTDB-Compaction-2] ERROR
> o.a.i.d.e.c.i.s.SizeTieredCompactionTask:196 - root.test.g_0-0 [Compaction]
> Throwable is caught during execution of SizeTieredCompaction, {}
> java.lang.NullPointerException: null
> at
> org.apache.iotdb.tsfile.read.TsFileSequenceReader.(TsFileSequenceReader.java:143)
> at
> org.apache.iotdb.tsfile.read.TsFileSequenceReader.(TsFileSequenceReader.java:123)
> at
> org.apache.iotdb.db.engine.compaction.inner.utils.MultiTsFileDeviceIterator.(MultiTsFileDeviceIterator.java:63)
> at
> org.apache.iotdb.db.engine.compaction.inner.utils.InnerSpaceCompactionUtils.compact(InnerSpaceCompactionUtils.java:71)
> at
> org.apache.iotdb.db.engine.compaction.inner.sizetiered.SizeTieredCompactionTask.doCompaction(SizeTieredCompactionTask.java:119)
> at
> org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.call(AbstractCompactionTask.java:68)
> at
> org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask.call(AbstractCompactionTask.java:45)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2022-03-25 17:01:47,597 [pool-7-IoTDB-Compaction-2] WARN
> o.a.i.d.e.c.i.s.SizeTieredCompactionTask:200 - root.test.g_0-0 [Compaction]
> Start to handle exception
> 2022-03-25 17:01:47,597 [pool-7-IoTDB-Compaction-2] INFO
> o.a.i.d.e.c.i.InnerSpaceCompactionExceptionHandler:85 - root.test.g_0-0
> [Compaction][ExceptionHandler] some source files [file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722177-3064-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722257-3067-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722295-3068-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722342-3070-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722378-3071-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722426-3073-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722472-3074-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722533-3076-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198722585-3077-0-1.tsfile,
> compactionCandidate: false, compacting: true, file is
> ./sbin/../data/data/sequence/root.test.g_0/0/0/1648198
[jira] [Assigned] (IOTDB-2818) An exception occurs when the weekly test is mixed with read and write
[ https://issues.apache.org/jira/browse/IOTDB-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reassigned IOTDB-2818: --- Sprint: 2022-3-Dragon Assignee: Liuxuxin (was: 周沛辰) > An exception occurs when the weekly test is mixed with read and write > - > > Key: IOTDB-2818 > URL: https://issues.apache.org/jira/browse/IOTDB-2818 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Server >Affects Versions: master branch >Reporter: xiaozhihong >Assignee: Liuxuxin >Priority: Major > Attachments: log_all.log, rw_no_overflow.config.properties > > > Sequence, when performing read-write mixing, two errors are encountered: > Negative Position and COMPACTION_CANDIDATE while its status is UNCLOSED. > Master Commit ID: 2022-03-28-12-47-46_5404730 > Sequential read and write mixing by using benchmark. > Please refer to the attachment for the configuration of the benchmark. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (IOTDB-2651) [compaction level-2] The write performance deteriorates severely
[ https://issues.apache.org/jira/browse/IOTDB-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504799#comment-17504799 ] Liuxuxin commented on IOTDB-2651: - The reason of this issue is the status flag `compactionCandidate` in TsFileResource. This flag indicates whether a file has been selected for the compaction task and the task is queued for execution. If a file is a compaction candidate, it will not be selected during the compaction selection. Under some situation, like failing to submit a compaction task to the queue, the compactionCandidate flag should have been set to false but was not. As a result, some files should be selected to be compacted cannot be selected because their compactionCandidate flag is true. > [compaction level-2] The write performance deteriorates severely > > > Key: IOTDB-2651 > URL: https://issues.apache.org/jira/browse/IOTDB-2651 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Compaction >Affects Versions: 0.13.0 >Reporter: 刘珍 >Assignee: Liuxuxin >Priority: Major > Labels: pull-request-available > Fix For: 0.13.0 > > Attachments: image-2022-03-02-17-47-06-196.png, > image-2022-03-02-17-50-45-149.png, image-2022-03-02-17-51-49-608.png, > image-2022-03-02-17-52-06-148.png, level-2_log_all.log.gz, screenshot-1.png, > screenshot-2.png > > > 测试版本master-pre3 > 测试环境：私有云2台，8C32G > 问题描述： > 长测配置，对齐序列，无乱序，运行5小时40分钟，发生合并2层文件，随之出现写入性能下降严重，同时GC上升。 > 测试结果： > 1. 写入性能，绿色的线： > !screenshot-2.png! > 2. master-pre3 printgc+开合并 Q7 GROUP BY语句按小时聚集求平均值-查询耗时 > !screenshot-1.png! > 3.磁盘IO > !image-2022-03-02-17-51-49-608.png! > 4.GC > !image-2022-03-02-17-52-06-148.png! > 5.log见附件 > 6.系统启动5小时40分钟，合并开始进行2层文件的合并，同时期显示gc次数上升，写入性能下降。因此推断合并2层文件导致了写入性能下降。 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (IOTDB-2503) Import a Status for TsFileResource
[ https://issues.apache.org/jira/browse/IOTDB-2503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reassigned IOTDB-2503: --- Assignee: Liuxuxin (was: Jack Tsai) > Import a Status for TsFileResource > -- > > Key: IOTDB-2503 > URL: https://issues.apache.org/jira/browse/IOTDB-2503 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Jialin Qiao >Assignee: Liuxuxin >Priority: Minor > Labels: pull-request-available > > Currently, we have isCompacting and isCandidate in TsFileResource for the > compaction process. > > It's better to add a status field in TsFileResource. This Status contains 4 > types: Compacting, CompactionCandidate, Unclosed, Closed. This field is to > replace the isClosed, compactionCandidate, Compacting. > > When Compacting, we only select the Unclosed TsFile to compact. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (IOTDB-2636) [compaction] Compaction affects query performance
[ https://issues.apache.org/jira/browse/IOTDB-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503957#comment-17503957 ] Liuxuxin commented on IOTDB-2636: - 在 IoTDB-Benchmark 中开启 debug 查询，比例为 1/1000，得到查询时的debug日志。对日志分析得到：benchmark查询的文件均为0层文件，没有 1 层文件。因此开启合并后查询性能下降可能是合并占用 CPU 和 IO 所致。 > [compaction] Compaction affects query performance > - > > Key: IOTDB-2636 > URL: https://issues.apache.org/jira/browse/IOTDB-2636 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Compaction >Affects Versions: 0.13.0-SNAPSHOT >Reporter: 刘珍 >Assignee: Liuxuxin >Priority: Critical > Attachments: image-2022-03-01-14-30-59-082.png, > image-2022-03-01-14-37-30-548.png, image-2022-03-01-14-44-37-559.png, > load-file.log > > > master-pre3 > 测试环境： > 私有云8C32G，16Gheap，无乱序，长测配置。 > 1. master 关合并/开合并对比 > cwt代表参数 merge_write_throughput_mb_per_sec > master开合并会导致查询性能下降约20%。 > !image-2022-03-01-14-30-59-082.png! > 2. master-pre3和0.12.5-pre6开合并对比 > merge_write_throughput_mb_per_sec均为8 > master总体比0.12慢70% > !image-2022-03-01-14-37-30-548.png! > 这个查询对应的写入性能（作为查询基于的数据量的参照），master查询的数据不大于0.12.5： > !image-2022-03-01-14-44-37-559.png! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (IOTDB-2651) [compaction level-2] The write performance deteriorates severely
[ https://issues.apache.org/jira/browse/IOTDB-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liuxuxin reassigned IOTDB-2651: --- Assignee: Liuxuxin > [compaction level-2] The write performance deteriorates severely > > > Key: IOTDB-2651 > URL: https://issues.apache.org/jira/browse/IOTDB-2651 > Project: Apache IoTDB > Issue Type: Bug > Components: Core/Compaction >Affects Versions: 0.13.0 >Reporter: 刘珍 >Assignee: Liuxuxin >Priority: Major > Attachments: image-2022-03-02-17-47-06-196.png, > image-2022-03-02-17-50-45-149.png, image-2022-03-02-17-51-49-608.png, > image-2022-03-02-17-52-06-148.png, level-2_log_all.log.gz > > > 测试版本master-pre3 > 测试环境：私有云2台，8C32G > 问题描述： > 长测配置，对齐序列，无乱序，运行5小时40分钟，发生合并2层文件，随之出现写入性能下降严重，同时GC上升。 > 测试结果： > 1. 写入性能，绿色的线： > !image-2022-03-02-17-47-06-196.png! > 2. 查询耗时 > !image-2022-03-02-17-50-45-149.png! > 3.磁盘IO > !image-2022-03-02-17-51-49-608.png! > 4.GC > !image-2022-03-02-17-52-06-148.png! > 5.log见附件 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (IOTDB-2653) Time range of TsFile resource overlaps after unseq compaction
Liuxuxin created IOTDB-2653: --- Summary: Time range of TsFile resource overlaps after unseq compaction Key: IOTDB-2653 URL: https://issues.apache.org/jira/browse/IOTDB-2653 Project: Apache IoTDB Issue Type: Bug Reporter: Liuxuxin Assignee: Liuxuxin Fix For: 0.13.0-SNAPSHOT Same as [IOTDB-2641|https://issues.apache.org/jira/browse/IOTDB-2641] -- This message was sent by Atlassian Jira (v8.20.1#820001)
